a discord bot for the tootsies server. ask, recap, discuss, ship features by typing.
CLAUDE.md’s Evals section carries the binding rules (two tiers, the GOLDEN mandate, gating-vs-trend); this file is the full catalog – what each eval grades, its judges, its golden, and the history that shaped it. Add an eval -> add its entry here (and its golden) in the same PR.
Two tiers, deliberately separate (see COVERAGE.md for the rule-by-rule matrix and gaps):
pytest suite plus utils/output_checks.py, a checker for the mechanically-verifiable prompt rules (em dash, emoji-unless-user-first, 200-char ceiling, two-sentence max, one-link-max, music TRACK line, first-person tool narration, memory-note hygiene). It’s mostly a checker module + tests only (not wired into the live send path), with ONE enforcement wired: strip_em_dashes runs on compose_market_drop’s output so no market surface (betting board / value alert / market drop / market alert) can SHIP an em dash even when the model ignores the prompt rule (owner report). Each predicate maps to a row in COVERAGE.md.scripts/eval_*.py on the scripts/evals/harness.py spine, aggregated by scripts/run_evals.py. They make live, paid, non-deterministic, model-judged calls (some web-grounded), so they run on a schedule / on demand, never blocking a merge — judge by trend, not a single verdict. Current evals: eval_ask_premise (fabrication), eval_ask_live_score (the context-relative sibling of eval_ask_premise: does /ask invent a FINAL for a still-LIVE game? The “this was horrible” incident — mid first half, with the verified live score (France 2-1) in her context AND on the message she was replying to, a bare “?” made her reply “final was France 4-0, Norway never scored, Dembele hat trick”; she fabricated a full-time result for a game ~24 min in and contradicted the 2-1 in her own context. Unlike the web-grounded premise judge this is CONTEXT-RELATIVE: it injects a real live MarketSnapshot + her prior live update and grades the answer against THAT in-context state, no web, so it’s rot-free and independent of how the match really ended; GOLDEN is the verbatim fabrication, which must fail; trend-only, graded on both models. NO prompt rule was added (the #849 discipline): the existing grounding already handles this on BOTH models — the legacy ask rule 9 (“never roll the live score forward to a later score or a final you didn’t pull”) + the constitution’s HOLD-VERIFIED-DATA-UNDER-PUSHBACK / NEVER-FABRICATE-scores — and a controlled n=5 measurement confirmed it (0/20 fabrications across Sonnet+Opus × the bare-“?” trigger + the explicit-“what was the final” probe). The prod incident was a nondeterministic one-off in a large (~48k-token) context, not a systematic gap. A per-model 9b/_OPUS_LIVE_SCORE rule + a room-post sharpen were drafted and reverted as the exact scar tissue docs/PROMPT_OPTIMIZATION.md warns against (don’t accrete a counter-rule for a one-off the existing text already covers; caving-to-a-“?” is a Sonnet sycophancy mode that Opus doesn’t even share). So the eval is the deliverable: a context-relative regression guard (golden teeth, both models) over a fabrication class no other eval covers, while the real-prod rate is monitored by eval_ask_premise’s web-grounded live-log pass), eval_ask_restraint (bartender restraint / staying in lane: does /ask co-sign a charged social claim handed to it — “isn’t this colorism”, “back me up” — as settled fact and argue the case at length, vs. give a bartender’s line and steer back? A SCOPE judge, distinct axis from the eval_ask_premise fabrication judge: an in-lane music/sports take PASSES, ratifying a contested identity claim as fact or writing the op-ed FAILS; GOLDEN is the verbatim shipped colorism take, which must fail; carries an in-lane control case so it can’t be satisfied by lobotomizing her into a non-answer machine; trend-only + live-log pass over real ask answers), eval_ask_tools (tool routing: does /ask pick lookup_catalog over web_search for a catalog question and answer right — wires the real catalog/reference handlers into client.ask since the cog, not the client, supplies tools; the chart-#2-count case (the #292 trigger) is a reference case now that #335 wired the authoritative lookup_reference source whose structured table extractor parses the Wikipedia discography chart, so a real source should flip it from “hedge” to “correct complete list”, with the wrong-list golden still hard-failing), eval_constitution (jailbreak/safety), eval_memory_fence (the fence; flagless it grades memory’s ENABLED model, --model <id> audits one rung — run on 2026-09-07 for #3072 across every Models-page rung: gpt-5.6-sol / gpt-5.6-terra / claude-fable-5-1 / gpt-6-astra all 7/7 with the golden flagged, Sonnet 5 control 6/7 on the same judge, the soft verbatim-quote nit), eval_memory_recall (conceptual recall, grades both paths: the keyword path’s write-side tagger + read-side query-expander meet in the middle (a “messiest thing” query’s expanded terms overlap a drama note’s keywords), AND the semantic-vector path — embeds note+query and checks cosine ≥ SIMILARITY_FLOOR, with the acceptance gate vector recall ≥ keyword baseline on the conceptual set; both keep the negative anchor that fails if recall over-broadens; metrics/anchor eval, deterministic given the model outputs, no DB needed, vector cases self-gate to informational without OPENAI_API_KEY), eval_image_intent (image nomination: grades whether the typed-@mention image surface tags correctly — an explicit “draw me X” always emits <image>, a plain factual question stays words, and with a source image present <remix> (edit) vs <image> (fresh) splits right; a deterministic tag check on client.ask(allow_image=True, allow_remix=...) output via parse_image_signal, so no paid image generation — one ask call per case, trend-only live cases with two hard-gating goldens (over-eager draw, remix-vs-generate)), eval_room_history (the server’s OWN past, #2182: does /ask retell the room’s history from its memory notes, or invent it? The judge is NOTE-GROUNDED, and that is the whole reason this eval had to exist separately – the room’s private lore is not on the web, so eval_ask_premise’s WEB-grounded judge structurally cannot check whether ‘the Jamaican caught a body’ happened, and the live-log pass grades QUESTION+ANSWER without the notes, so an invented room event is invisible to it too. This judge is handed the SAME memory block the model got (injected through the real client.ask(memory_context=...) path, so it grades the shipped pipeline) and fails an answer asserting any room event those notes do not support. A closed-world check, sound only BECAUSE the corpus is closed: the notes are the entire record, so ‘not in the notes’ really does mean ‘not established’. GOLDEN is the verbatim 2026-08-08 season-1 answer that started #2182 (‘the jamaican caught a body’, ‘you and MARTINI were the villains’) – neither event is in any note, so a judge that passes it cannot detect the class at all. Measured at ship 2026-08-31: golden caught, 4/4 live scenarios pass – which is the first evidence that #2181’s dated-notes + _OPUS_MEMORY fix actually holds, since nothing could grade it before – and 5/5 on a DISCRIMINATION set that is the real test of the judge, not the blatant golden: it fails an answer that is fully grounded except for ONE invented member action (‘sailingsan got banned for a week’) or ONE invented outcome (‘MARTINI won that argument’), while passing slang, attitude, and an honest ‘i don’t have much on that’. Admitting the limit is a PASS by design – the failure mode is filling the gap, not having one. Trend-only; the golden hard-gates), eval_music_post (voice + the ENGINEERED ONE-LINER, #commentary-tone: the judge grades a drop on SHAPE, so it fails the imagined scene (“made this for when the lights come up”), the abstract transformation (“turning X into Y is nasty work”), and the tacked-on verdict (“, that hook still knows the assignment”) while a metaphor on its own still passes – the persona is figurative on purpose. It carries a SECOND golden: the verbatim 2026-09-13 Doechii drop the owner flagged, which is short, casual, grounded and names a real artist, so it clears every test the original judge ran. A judge that passes it is blind to the whole class, live-log pass included. Both goldens were caught on every dry-run pass; the live cases stay trend-only because per-line verdicts on borderline drops move run to run), eval_discourse_post / eval_chimein_post / eval_icebreaker (room-surface behavior + grounding). eval_icebreaker carries two judges: a per-opener one (concrete/answerable, room-directed, linkless, not dead-air, doesn’t pick the side for them) AND a diversity judge graded over a SET of recent openers that fails on template repetition (the “most overrated X” run) — the one thing per-post scoring and text dedup structurally can’t catch. eval_gif_pick (gif vision-pick taste: a live, vision-judged behavioral eval that runs real Giphy searches for a set of reactions, lets claude.pick_gif choose, and grades the chosen gif’s preview frame for fit — reads as the reaction, fits the moment’s tone, lands as her sharp/ironic voice not a corny one; the search query IS the reaction as in prod so it measures pick QUALITY, not the skip gate; its GOLDEN is a celebration gif graded against a grief reaction that the judge MUST fail — judge-teeth, rot-free by reusing a fetched candidate; self-gates to informational when GIPHY_API_KEY is unset). eval_art_resolution (the image-resolution LADDER, #thin-ahead-art: a headline / a Kalshi market’s own fields in, the SOURCE RUNG that answered out — tmdb_poster/omdb_poster for a film, deezer_artist for a musician, wiki_image for a place/thing/org, the subtitle-read catalog hint for a music market, none when nothing authoritative exists. Asserting the RUNG (not the bytes) keeps it stable as catalogs change, and the cases are the five REAL production bugs from that day — a film named mid-essay, a governing body, the Deezer placeholder, the wrong-artist market, and an abstract topic that must resolve NOTHING (the invented-art inverse). No judge to drift (a string comparison), but trend_only because every rung is a live third-party fetch — an observed run had two cases transiently resolve none and pass on retry, so a no-art result retries once (retry_async result-predicate) before scoring. GOLDEN: the “I’m The Problem” market asserted to resolve the WRONG artist (Lizzo) — it must FAIL, and a pass means the wrong-artist path is back or the checker lost its teeth. #youtube-chart-art added a live YOUTUBE-CHART case: the video leading kworb’s most-viewed chart right now becomes a Kalshi music-video market’s leading leg, and the card must come back with that video’s own thumbnail (source=youtube_chart) — graded against the live chart, so it can’t rot. Its GOLDEN asserts that a non-word SUBSTRING (‘peta’) matches a chart title (‘Ariana Grande - petal’) — it must FAIL, because the whole-phrase test is the one thing keeping short leg labels from pulling in unrelated videos). eval_source_photo (the SOURCE-photo gate, #source-media-reuse: given a moment and the photo the WIRE attached to it, is that photo TAKEN or thrown away? eval_art_resolution grades which RUNG answered and structurally can’t see this — a stock TMDB portrait is a perfectly legitimate rung, it’s the right source for the wrong picture — so this grades the one axis that was failing. Cases are real wire posts in BOTH directions: four TAKE cases where the subject isn’t face-recognizable (a clip still in full prosthetics, a masked character still, an athlete shot wide, a real photo under a “HERE WE GO” text overlay — exactly the class the identity framing was killing) and five REJECT cases (a rankings table over a photo, a lineup poster, a revenue chart, a bare logo, plus a real photo attached to a plainly DIFFERENT moment — the guard against the fix degrading into “take whatever is attached”). Take-vs-reject against a labelled expectation, so there’s no second judge to drift; each case runs 3× and scores by majority so one flake can’t flip a verdict; trend_only since the gate is a model and the images are live X CDN fetches. GOLDEN: the Blanka clip still asserted to be REJECTED — the exact behaviour that shipped the stock red-carpet portrait; it must FAIL, and a pass means the identity framing is back on the source path). eval_curator_pick (the curator TEXT fit-judge, #curator: does pick_curator_text pick the on-theme candidate for a link/text channel (#man: cars/menswear) over off-theme noise — spam, wrong topic — and SKIP when nothing fits? Deterministic-by-construction (exactly one on-theme candidate per case, so the pick INDEX is the verdict, no second judge call); GOLDEN is a lone off-theme post a healthy judge must skip, so a pick = toothless; trend-only, needs only ANTHROPIC_API_KEY. The vision half shares the pick_gif/pick_curator path eval_gif_pick already covers). eval_wire_brief (the wire desks’ CONFIRMED-BRIEF same-moment discipline, #wrong-person: through the real compose_pop_take, the shipped mismatch case — the VERBATIM re-run Perplexity brief that glued the Larissa Nery voter-roll story onto @DailyLoud’s unnamed “Brazilian model is going viral” clip — must NOT leak the substituted story (EMPTY passes; deterministic string check, no second judge), plus two same-moment-brief controls that must still USE the brief (the over-suppression axis). Promoted from the one-shot dryrun_wire_brief_mismatch A/B so drift is caught on the eval schedule; trend-only (the compose is a model; measured at ship: gpt leaked 5/5 old framing → 0/5 fixed, sonnet 2/5 → 0/5). GOLDEN: the verbatim shipped Larissa Nery take graded by the same leak check — it must FAIL, else the detector lost the tokens that catch the substitution). eval_art_resolution also carries the #wrong-person additions: an artist-centered take (“Dua Lipa’s Radical Optimism tour…”) must resolve HER deezer_artist photo (the deleted bespoke path fed the name to a song-first title search and Jack Harlow’s SONG “Dua Lipa” won it), and a GOLDEN asserting an artist-less music title resolves a cover — it must FAIL, else the resolve_catalog_art no-artist hardening is gone. eval_ai_register (the DETERMINISTIC AI-register pass, #ai-register: scores text for the MECHANICAL “sounds like AI” tells via utils/slop_score.py — the vendored sam-paech/antislop lexicon (MIT, 495 terms after pruning fiction-corpus names; validated ZERO false fires on 223 real owner-hand-posted X replies) + a not-X-but-Y contrast-construction regex battery — then scores every REAL shipped post the live-logs pull surfaces, naming the exact tells (score=0.4 tells: testament to, isn't just X, it's Y) as trend findings. No judge spend: fully deterministic, the free complement to the golden-anchored REGISTER judge (dryrun_x_reply_voice.py), which alone can catch the engineered-one-liner shapes (mirrored clauses, abstract-metaphor verdicts) that carry no lexicon word. GOLDENS: a slop paragraph + a contrast one-liner must flag, the persona voice examples must stay clean — the same fixtures gate deterministically in Tier 1 (tests/test_slop_score.py), so the two layers can’t drift apart. eval_perplexity_params is a standalone informational tuning sweep, not registered in run_evals.eval_discourse_post, eval_music_post, eval_chimein_post, eval_icebreaker) plus eval_ask_premise don’t only grade synthetic posts — each also pulls the real shipped posts for its surface from the last window of Railway EVENT logs (the post_preview fields) and grades every one through its same JUDGE, via scripts/evals/live_logs.py:live_cases. (eval_ask_premise grades real /ask answers from the ask_answered event answer-only, since the asker’s question isn’t logged — added in #315 to give the surface the #292 chart-fact bug lived on real production observability.) So run_evals.py’s report — and the deduped auto-eval issues the evals.yml workflow files off it — cover what the bot actually posted, not just what it posts for a fixture. The pull is best-effort and gated on RAILWAY_API_TOKEN/RAILWAY_SERVICE_ID: without them (or on any fetch error) the live cases are simply empty and the suite runs in its existing API-only mode, so the pass is purely additive. It reuses ops_monitor.fetch_events (one memoized pull per suite run, same window as the ops report) and caps grading at MAX_LIVE_PER_SURFACE newest posts to bound spend. Live cases are non-golden and named live[surface]: … so a failing production post turns the eval red and the issue judge can tell production samples from fixtures. Only surfaces whose output lands in the logs as a preview get a live pass (recaps still aren’t logged verbatim, by data minimization; /ask now logs the real QUESTION+ANSWER pair — her full answer plus the asker’s prompt — so the fabrication judge grades production answers with the same context the synthetic cases get). The preview must exceed the post ceiling, or the pass measures nothing (#preview-truncation): these previews were capped at 120 chars while room posts run to a 200-char ceiling, so a normal two-sentence take reached the judge CUT MID-SENTENCE and was failed for being a fragment with no claim — true of any fragment, however good the post was. The live discourse pass read 1/10 with failures literally reading “the post cuts off mid-sentence”, and the ONE pass was the single take short enough (118 chars) to arrive whole. A measured full-vs-truncated A/B through the SAME judge (scripts/dryrun_preview_truncation.py) put the identical posts at 14/15 whole vs 3/15 truncated, every over-cap post flipping 3/3 → 0/3 — so the surface was healthy (~93%) and the number was an artifact. /ask had already learned this (#315, it logs its answer untruncated so “a late hedge isn’t clipped into a false-fail”); the fix simply hadn’t reached the other five graded surfaces. Now ONE shared utils.events.POST_PREVIEW_CHARS (400, clearing the 200 ceiling with room for a trailing link) + post_preview() feeds every graded surface, guarded by tests/test_events.py:test_post_preview_clears_the_room_post_ceiling so lowering it back under the ceiling fails CI. The standing lesson is docs/PROMPT_OPTIMIZATION.md’s: VALIDATE THE METRIC before acting on it — a bad number here would have bought prompt scar tissue on a surface that was fine. The pure collect_posts + surface map are unit-tested in tests/test_evals_live_logs.py; the Railway fetch is integration-only.ANTHROPIC_API_KEY=... python scripts/run_evals.py # whole suite
python scripts/run_evals.py --filter constitution --out r.md # one eval + report file
python scripts/eval_ask_premise.py # a single eval standalone
A judge-model upgrade can leak a golden, and two different defects cause it (#3131). The scheduled suite went red for eight days after #3029 moved the judges from claude-sonnet-4-6 to claude-sonnet-5. Two goldens stopped failing. The judges did not get weaker. Each leak had its own cause, and the two causes need opposite fixes.
chimein_post :: the judge could not see the context its own rule needed. The ON POINT rule asks if the post rebuts a claim nobody made. _grade sent the judge only the reply, never the chat. The judge therefore invented a chat that fit the reply, and it passed the point-hijack golden. Its stated reasons named the invention directly: “reacting to a chat consensus”, “an implied chat argument”. The fix gives the judge the chat. Each golden now carries the chat it answered. A live-log preview has no chat (the buffer is not logged, by data minimization), so it passes (not shown) and the judge skips that one rule instead of guessing. eval_commentate already used this shape with its GROUND TRUTH frame. A sweep of the other judges found no third case.opus_discourse :: the golden never matched the failure it was named for. The fixture ended “regular season MVP is the prettiest consolation prize in sports”. That is an arguable take, which is what rule 1 asks FOR, so a literal reading of the rubric passes it. The old judge failed it for the wrong reason. The verdict now closes its own question out loud, so it is the closed loop the name claims.Fix the fixture before you touch the rubric. A sharpened rubric was measured first and rejected. It did catch the golden, but it also flipped two healthy shipped posts from pass to fail. That moves the bar the surface is graded on, which a golden repair must never do. The live pass rate held at 3/4 across the fix, which is the check that proves the bar did not move. So: measure a candidate fix against the goldens AND the real posts before shipping it. A golden must also fail RELIABLY. One candidate leaked once in five runs and was dropped for one that failed five times out of five.
Every eval carries a GOLDEN fixture — a known-bad input that MUST come back failing. A judge is itself a model and drifts; a golden that stops failing is declared toothless and counts as a regression (harness.exit_code). When you add an eval, add a golden. The harness’s deterministic core (parse_judge_json, exit-code/regression logic) is unit-tested in tests/test_evals_harness.py; the eval wiring is guarded no-API by tests/test_eval_wiring.py.
Gating vs trend (EvalResult.trend_only). The job (run_evals exit code) goes red only on a gating regression: a leaked golden, a deterministic classifier-accuracy gate breaching its floor (classify_*, preflight_order, chimein_score, discourse_score), or a crash. The model-judge-on-generated-text behavioral evals (constitution, memory_fence, ask_premise, ask_tools, music_post, discourse_post, chimein_post, icebreaker, recap) are marked trend_only=True: their GOLDENS still hard-gate (the judge must keep its teeth), but a single nondeterministic live-case FAIL is reported (tagged [trend-fail], surfaced in the overall line as a “trend finding”) for evals.yml’s judge to file on by trend — it does not red the job. This matches the suite’s “NOT a CI gate, judged by trend, a single live-case FAIL can be noise” contract: the job’s red/green tracks the trustworthy signals, while behavioral fabrication/voice drift still reaches the judge. parse_judge_json also recovers a clean verdict when the judge wraps its explanation in stray unescaped quotes, so a valid pass isn’t scored as a spurious FAIL.