Skip to content

Golden paths

Each directory here is a use case you can run from a clean clone: a README with the same seven sections, a run.sh that runs the README's own commands, and its inputs on disk. tests/test_golden_paths.py runs every path whose prerequisite is none and pins one line of its output, so the docs are what the suite asserts.

path what it shows prerequisite runtime
agents/ the install doctor's report, the per-surface agent configs attest emit generates, and one AGENT_SURFACES surface driven over stdio the way hermes-agent actually drives it none — pure local computation ~10 s
citations/ a four-entry BibTeX library written by bibtexparser, a draft citing three real keys and one that resolves nowhere, and the citation lint that cite.check/runs.claims_check over MCP and attest claims from the CLI both run none — pure local computation ~4 s
dvc/ the run ledger reading a real dvc repro pipeline — four foreach sweep arms ranked, and dvc.lock's own recorded param list read apart from each arm's actual value, with no dependency on the dvc package none — pure local computation ~1 s
flows/ forty labelled items scored for two personas, every MCP tool driven over stdio, four MLflow arms read back by the ledger none — pure local computation ~75 s
hydra/ the run ledger reading a real Hydra --multirun sweep — four sweep arms ranked, and the finding that hydra.job.chdir=True is required or all four arms silently overwrite one shared metrics.json none — pure local computation ~1 s
mlflow/ the run ledger reading a real MLflow directory — four sweep arms ranked, one claim checked against them and contradicted on purpose, the same arms retrained in about two seconds none — pure local computation ~3 s
model-servers/ attest ingest and attest tag run as real subprocesses against an in-process OpenAI-compatible stub, plus a table for pointing LLM_BASE_URL at vLLM/llama.cpp/LM Studio/Ollama none — pure local computation ~2 s
ranking/ twenty hand-built rows ranked with rank.rank_rows, no database — the classifier-only AUC on six labelled clicks beside the rank-order AUC of the blended list, the two numbers evaluate_user's docstring says are different none — pure local computation ~1 s
sacred/ the run ledger reading a real Sacred FileStorageObserver directory — four sweep arms ranked, and run.json's own result field read as a metric alongside metrics.json's logged series none — pure local computation ~1 s
tensorflow/ the run ledger reading a real Keras training run through CSVLogger's CSV and a plain metrics JSON — four learning-rate arms ranked, and the family_of fix a bare lr_<value> stem required none — pure local computation ~1 s
wandb/ the run ledger reading a real W&B offline directory — four sweep arms ranked, and the finding that offline W&B never writes its own summary/config files until synced none — pure local computation ~1 s
workspace/ the run ledger and claim checker over a two-project workspace with three claims deliberately wrong none — pure local computation ~1 s
prompt-evals/ a score for the shipping tagging prompt on 28 dev cases, and the transfer gate that decided whether a GEPA candidate could replace it a model server at LLM_BASE_URL ~2 min