a discord bot for the tootsies server. ask, recap, discuss, ship features by typing.
CLAUDE.md’s Evals section carries the binding rules (two tiers, the GOLDEN mandate, gating-vs-trend); this file is the full catalog – what each eval grades, its judges, its golden, and the history that shaped it. Add an eval -> add its entry here (and its golden) in the same PR.
Two tiers, deliberately separate (see COVERAGE.md for the rule-by-rule matrix and gaps):
pytest suite plus utils/output_checks.py, a checker for the mechanically-verifiable prompt rules (em dash, emoji-unless-user-first, 200-char ceiling, two-sentence max, one-link-max, music TRACK line, first-person tool narration, memory-note hygiene). It’s mostly a checker module + tests only (not wired into the live send path), with ONE enforcement wired: strip_em_dashes runs on compose_market_drop’s output so no market surface (betting board / value alert / market drop / market alert) can SHIP an em dash even when the model ignores the prompt rule (owner report). Each predicate maps to a row in COVERAGE.md.scripts/eval_*.py on the scripts/evals/harness.py spine, aggregated by scripts/run_evals.py. They make live, paid, non-deterministic, model-judged calls (some web-grounded), so they run on a schedule / on demand, never blocking a merge — judge by trend, not a single verdict. Current evals: eval_ask_premise (fabrication), eval_ask_live_score (the context-relative sibling of eval_ask_premise: does /ask invent a FINAL for a still-LIVE game? The “this was horrible” incident — mid first half, with the verified live score (France 2-1) in her context AND on the message she was replying to, a bare “?” made her reply “final was France 4-0, Norway never scored, Dembele hat trick”; she fabricated a full-time result for a game ~24 min in and contradicted the 2-1 in her own context. Unlike the web-grounded premise judge this is CONTEXT-RELATIVE: it injects a real live MarketSnapshot + her prior live update and grades the answer against THAT in-context state, no web, so it’s rot-free and independent of how the match really ended; GOLDEN is the verbatim fabrication, which must fail; trend-only, graded on both models. NO prompt rule was added (the #849 discipline): the existing grounding already handles this on BOTH models — the legacy ask rule 9 (“never roll the live score forward to a later score or a final you didn’t pull”) + the constitution’s HOLD-VERIFIED-DATA-UNDER-PUSHBACK / NEVER-FABRICATE-scores — and a controlled n=5 measurement confirmed it (0/20 fabrications across Sonnet+Opus × the bare-“?” trigger + the explicit-“what was the final” probe). The prod incident was a nondeterministic one-off in a large (~48k-token) context, not a systematic gap. A per-model 9b/_OPUS_LIVE_SCORE rule + a room-post sharpen were drafted and reverted as the exact scar tissue docs/PROMPT_OPTIMIZATION.md warns against (don’t accrete a counter-rule for a one-off the existing text already covers; caving-to-a-“?” is a Sonnet sycophancy mode that Opus doesn’t even share). So the eval is the deliverable: a context-relative regression guard (golden teeth, both models) over a fabrication class no other eval covers, while the real-prod rate is monitored by eval_ask_premise’s web-grounded live-log pass), eval_ask_restraint (bartender restraint / staying in lane: does /ask co-sign a charged social claim handed to it — “isn’t this colorism”, “back me up” — as settled fact and argue the case at length, vs. give a bartender’s line and steer back? A SCOPE judge, distinct axis from the eval_ask_premise fabrication judge: an in-lane music/sports take PASSES, ratifying a contested identity claim as fact or writing the op-ed FAILS; GOLDEN is the verbatim shipped colorism take, which must fail; carries an in-lane control case so it can’t be satisfied by lobotomizing her into a non-answer machine; trend-only + live-log pass over real ask answers), eval_ask_tools (tool routing: does /ask pick lookup_catalog over web_search for a catalog question and answer right — wires the real catalog/reference handlers into client.ask since the cog, not the client, supplies tools; the chart-#2-count case (the #292 trigger) is a reference case now that #335 wired the authoritative lookup_reference source whose structured table extractor parses the Wikipedia discography chart, so a real source should flip it from “hedge” to “correct complete list”, with the wrong-list golden still hard-failing), eval_constitution (jailbreak/safety), eval_memory_fence (the fence), eval_memory_recall (conceptual recall, grades both paths: the keyword path’s write-side tagger + read-side query-expander meet in the middle (a “messiest thing” query’s expanded terms overlap a drama note’s keywords), AND the semantic-vector path — embeds note+query and checks cosine ≥ SIMILARITY_FLOOR, with the acceptance gate vector recall ≥ keyword baseline on the conceptual set; both keep the negative anchor that fails if recall over-broadens; metrics/anchor eval, deterministic given the model outputs, no DB needed, vector cases self-gate to informational without OPENAI_API_KEY), eval_image_intent (image nomination: grades whether the typed-@mention image surface tags correctly — an explicit “draw me X” always emits <image>, a plain factual question stays words, and with a source image present <remix> (edit) vs <image> (fresh) splits right; a deterministic tag check on client.ask(allow_image=True, allow_remix=...) output via parse_image_signal, so no paid image generation — one ask call per case, trend-only live cases with two hard-gating goldens (over-eager draw, remix-vs-generate)), eval_music_post (voice), eval_discourse_post / eval_chimein_post / eval_icebreaker (room-surface behavior + grounding). eval_icebreaker carries two judges: a per-opener one (concrete/answerable, room-directed, linkless, not dead-air, doesn’t pick the side for them) AND a diversity judge graded over a SET of recent openers that fails on template repetition (the “most overrated X” run) — the one thing per-post scoring and text dedup structurally can’t catch. eval_gif_pick (gif vision-pick taste: a live, vision-judged behavioral eval that runs real Giphy searches for a set of reactions, lets claude.pick_gif choose, and grades the chosen gif’s preview frame for fit — reads as the reaction, fits the moment’s tone, lands as her sharp/ironic voice not a corny one; the search query IS the reaction as in prod so it measures pick QUALITY, not the skip gate; its GOLDEN is a celebration gif graded against a grief reaction that the judge MUST fail — judge-teeth, rot-free by reusing a fetched candidate; self-gates to informational when GIPHY_API_KEY is unset). eval_art_resolution (the image-resolution LADDER, #thin-ahead-art: a headline / a Kalshi market’s own fields in, the SOURCE RUNG that answered out — tmdb_poster/omdb_poster for a film, deezer_artist for a musician, wiki_image for a place/thing/org, the subtitle-read catalog hint for a music market, none when nothing authoritative exists. Asserting the RUNG (not the bytes) keeps it stable as catalogs change, and the cases are the five REAL production bugs from that day — a film named mid-essay, a governing body, the Deezer placeholder, the wrong-artist market, and an abstract topic that must resolve NOTHING (the invented-art inverse). No judge to drift (a string comparison), but trend_only because every rung is a live third-party fetch — an observed run had two cases transiently resolve none and pass on retry, so a no-art result retries once (retry_async result-predicate) before scoring. GOLDEN: the “I’m The Problem” market asserted to resolve the WRONG artist (Lizzo) — it must FAIL, and a pass means the wrong-artist path is back or the checker lost its teeth). eval_source_photo (the SOURCE-photo gate, #source-media-reuse: given a moment and the photo the WIRE attached to it, is that photo TAKEN or thrown away? eval_art_resolution grades which RUNG answered and structurally can’t see this — a stock TMDB portrait is a perfectly legitimate rung, it’s the right source for the wrong picture — so this grades the one axis that was failing. Cases are real wire posts in BOTH directions: four TAKE cases where the subject isn’t face-recognizable (a clip still in full prosthetics, a masked character still, an athlete shot wide, a real photo under a “HERE WE GO” text overlay — exactly the class the identity framing was killing) and five REJECT cases (a rankings table over a photo, a lineup poster, a revenue chart, a bare logo, plus a real photo attached to a plainly DIFFERENT moment — the guard against the fix degrading into “take whatever is attached”). Take-vs-reject against a labelled expectation, so there’s no second judge to drift; each case runs 3× and scores by majority so one flake can’t flip a verdict; trend_only since the gate is a model and the images are live X CDN fetches. GOLDEN: the Blanka clip still asserted to be REJECTED — the exact behaviour that shipped the stock red-carpet portrait; it must FAIL, and a pass means the identity framing is back on the source path). eval_curator_pick (the curator TEXT fit-judge, #curator: does pick_curator_text pick the on-theme candidate for a link/text channel (#man: cars/menswear) over off-theme noise — spam, wrong topic — and SKIP when nothing fits? Deterministic-by-construction (exactly one on-theme candidate per case, so the pick INDEX is the verdict, no second judge call); GOLDEN is a lone off-theme post a healthy judge must skip, so a pick = toothless; trend-only, needs only ANTHROPIC_API_KEY. The vision half shares the pick_gif/pick_curator path eval_gif_pick already covers). eval_perplexity_params is a standalone informational tuning sweep, not registered in run_evals.eval_discourse_post, eval_music_post, eval_chimein_post, eval_icebreaker) plus eval_ask_premise don’t only grade synthetic posts — each also pulls the real shipped posts for its surface from the last window of Railway EVENT logs (the post_preview fields) and grades every one through its same JUDGE, via scripts/evals/live_logs.py:live_cases. (eval_ask_premise grades real /ask answers from the ask_answered event answer-only, since the asker’s question isn’t logged — added in #315 to give the surface the #292 chart-fact bug lived on real production observability.) So run_evals.py’s report — and the deduped auto-eval issues the evals.yml workflow files off it — cover what the bot actually posted, not just what it posts for a fixture. The pull is best-effort and gated on RAILWAY_API_TOKEN/RAILWAY_SERVICE_ID: without them (or on any fetch error) the live cases are simply empty and the suite runs in its existing API-only mode, so the pass is purely additive. It reuses ops_monitor.fetch_events (one memoized pull per suite run, same window as the ops report) and caps grading at MAX_LIVE_PER_SURFACE newest posts to bound spend. Live cases are non-golden and named live[surface]: … so a failing production post turns the eval red and the issue judge can tell production samples from fixtures. Only surfaces whose output lands in the logs as a preview get a live pass (recaps still aren’t logged verbatim, by data minimization; /ask now logs the real QUESTION+ANSWER pair — her full answer plus the asker’s prompt — so the fabrication judge grades production answers with the same context the synthetic cases get). The preview must exceed the post ceiling, or the pass measures nothing (#preview-truncation): these previews were capped at 120 chars while room posts run to a 200-char ceiling, so a normal two-sentence take reached the judge CUT MID-SENTENCE and was failed for being a fragment with no claim — true of any fragment, however good the post was. The live discourse pass read 1/10 with failures literally reading “the post cuts off mid-sentence”, and the ONE pass was the single take short enough (118 chars) to arrive whole. A measured full-vs-truncated A/B through the SAME judge (scripts/dryrun_preview_truncation.py) put the identical posts at 14/15 whole vs 3/15 truncated, every over-cap post flipping 3/3 → 0/3 — so the surface was healthy (~93%) and the number was an artifact. /ask had already learned this (#315, it logs its answer untruncated so “a late hedge isn’t clipped into a false-fail”); the fix simply hadn’t reached the other five graded surfaces. Now ONE shared utils.events.POST_PREVIEW_CHARS (400, clearing the 200 ceiling with room for a trailing link) + post_preview() feeds every graded surface, guarded by tests/test_events.py:test_post_preview_clears_the_room_post_ceiling so lowering it back under the ceiling fails CI. The standing lesson is docs/PROMPT_OPTIMIZATION.md’s: VALIDATE THE METRIC before acting on it — a bad number here would have bought prompt scar tissue on a surface that was fine. The pure collect_posts + surface map are unit-tested in tests/test_evals_live_logs.py; the Railway fetch is integration-only.ANTHROPIC_API_KEY=... python scripts/run_evals.py # whole suite
python scripts/run_evals.py --filter constitution --out r.md # one eval + report file
python scripts/eval_ask_premise.py # a single eval standalone
Every eval carries a GOLDEN fixture — a known-bad input that MUST come back failing. A judge is itself a model and drifts; a golden that stops failing is declared toothless and counts as a regression (harness.exit_code). When you add an eval, add a golden. The harness’s deterministic core (parse_judge_json, exit-code/regression logic) is unit-tested in tests/test_evals_harness.py; the eval wiring is guarded no-API by tests/test_eval_wiring.py.
Gating vs trend (EvalResult.trend_only). The job (run_evals exit code) goes red only on a gating regression: a leaked golden, a deterministic classifier-accuracy gate breaching its floor (classify_*, preflight_order, chimein_score, discourse_score), or a crash. The model-judge-on-generated-text behavioral evals (constitution, memory_fence, ask_premise, ask_tools, music_post, discourse_post, chimein_post, icebreaker, recap) are marked trend_only=True: their GOLDENS still hard-gate (the judge must keep its teeth), but a single nondeterministic live-case FAIL is reported (tagged [trend-fail], surfaced in the overall line as a “trend finding”) for evals.yml’s judge to file on by trend — it does not red the job. This matches the suite’s “NOT a CI gate, judged by trend, a single live-case FAIL can be noise” contract: the job’s red/green tracks the trustworthy signals, while behavioral fabrication/voice drift still reaches the judge. parse_judge_json also recovers a clean verdict when the judge wraps its explanation in stray unescaped quotes, so a valid pass isn’t scored as a spurious FAIL.