simulate

Synthetic reader reactions, so the ranker has negatives to learn from.

The classifier needs both classes. classifier_probs returns None on a single-class history, so a persona with only positive feedback ranks by embedding similarity alone forever -- which is the state every real account in this database is in. Nothing marks items not useful, because saying "no" to a recommendation is work nobody does.

Why not reuse bootstrap_persona. It labels by a linear threshold on the same embedding the classifier trains on, so a classifier fit on X to predict a threshold of X recovers it perfectly. Its own docstring says so, and evaluate_user excludes source='bootstrap' for exactly that reason. Labels generated here must be independent of the embedding or they rebuild that tautology in a new costume.

So a chat model reads the title and abstract and reacts as the persona would, in prose, then commits to a verdict. That judgement runs on text, through a different model, with no access to the vector -- which is what makes the resulting rows worth training on.

These are still simulated readers, not real ones. Rows are written with source='simulated' so they can be told apart from ui and agent clicks forever, and so a future evaluation can exclude them the way evaluate_user excludes bootstrap rows.

Reaction

Bases: BaseModel

A reader's response to one item.

reasoning comes before verdict on purpose: a small model asked for a bare label picks one and rationalises afterwards, while one asked to react first commits to something it then has to be consistent with. It is also the part a human can audit -- a verdict with no reasoning is unreviewable.

classifier_would_train(conn, user_id)

Whether this user's history has both classes, so the classifier fires.

The one question simulation exists to turn from no into yes.

react_to_item(chat_fn, persona, interests, title, summary)

One simulated reaction. Raises on an unusable response.

No retry and no fallback verdict: an unparseable reply must not become a silent useful=False, which would poison training data with the model's formatting problems rather than its judgement.

reaction_messages(persona, interests, title, summary)

The ONE renderer of the reaction prompt.

react_to_item and evals/reaction_eval.py both call this, so a score is always a score of the prompt feed.simulate_ratings actually sends.

simulate_feedback(conn, chat_fn, user_name, items, *, min_confidence=3)

Record a simulated reaction per item. Returns what was written and why.

Unsure verdicts are dropped rather than recorded, since a coin-flip recorded as a label is noise the classifier then has to fit.

The field is confidence, not strength, because the first version asked for "how strongly you feel" and the model read that as enthusiasm: a correct, well-argued rejection of a sourdough recipe came back at strength 1 and was filtered out as indifference. Every negative was discarded and the classifier still could not train -- the exact failure this module exists to fix, reintroduced by the word used to ask.

source_skew_caveat(conn, user_id)

Warn when the two classes are separable by feed rather than by topic.

Sampling round-robin across feeds is what finally produced negatives, but it has a cost: on matt's history it left 42 of 45 positives in arXiv cs.LG while negatives spread over nine other sources. A classifier fit on that can score a perfect AUC by learning "cs.LG means useful" -- which the embedding encodes trivially -- and learn nothing about the reader.

An AUC of 1.0 usually means the task was easy, not that the model is good. Saying so with the result is cheaper than someone trusting it.