tootsies

a discord bot for the tootsies server. ask, recap, discuss, ship features by typing.

---

Project maintained by mejasonmejason Hosted on GitHub Pages — Theme by mattgraham

Optimizing prompts for Opus: cut and reword, don’t add

The thesis of Epic #849. Read this before you touch any text Opus sees.

Every piece of text the model reads is both a cost and a lever: the constitution, the persona, each per-surface prompt, every tool description, every injected context-block label, even the eval judges. The default instinct when a model misbehaves is to add a rule to correct it. For Opus that instinct is almost always wrong. The right move is to find the existing text that is causing the behavior and cut or reword it.

Why “add a rule” fails on Opus specifically

The shared prompt was tuned, fix by fix, against Sonnet’s failure modes (sycophancy, hedging, stiffness, fence-sitting). Opus does not share those failure modes, and it differs in two ways that make accretion actively harmful:

1. Opus follows instructions literally. A reductive verb it’s handed (“Compact”, “Synthesize”, “Drop the noise”) it obeys — it compacts. Sonnet largely ignores those words. So a phrase that is inert on Sonnet is a live instruction on Opus.
2. Opus over-commits. Hand it a maximal instruction (“be exhaustive”, “the complete record”, “breadth so any question has a hit”) and it maximizes — it pads. A counter-rule stacked on top doesn’t cancel the first; now it’s following both, harder.

So when Opus drifts, adding a counter-instruction compounds the problem. It also bloats the prompt, and a longer prompt is itself a cost (tokens, latency, and more surface for cross-rule contradiction).

The method (what to do instead)

1. Find the cause in existing text; remove or reword it. Don’t write a new rule. Ask: what am I already saying that produces this?
2. Show before/after verbatim, every edit. No silent rewrites. The diff is the unit of review.
3. Dry-run on REAL data, not synthetic. Pull production logs/notes (Railway, /debug/query) and run the actual pipeline. A fixture can’t show you what the live distribution does.
4. Measure the axis you actually care about, with the real machinery. For recall that means max(prose_vector, tags_vector) cosine over the real query set — not a vibe read of the text. For length, the production answer_length telemetry. Name the metric before you edit.
5. Ablate: cut vs add vs reword, and measure each. (See #858’s ablation table — cutting the regulars block made the symptom worse; an explicit clause fixed it; the real lever was a layer up.) The first thing you blame is usually not the cause.
6. Length is not the enemy — bloat-without-payoff is. Don’t trim for its own sake. Measure whether the extra text buys the thing (searchability, depth, accuracy). If it doesn’t, it’s pure context cost and should go.
7. Key everything per model; share only the safety floor. A rule earned against Sonnet’s failures mis-fires on Opus. The constitution HARD rules, the memory fence, and protected-path/preflight guarantees are the one universal base and are never loosened per-model. Everything else is per-model.

Worked examples (this epic)

• A rule outlives its cause and starts fighting the surface (#3363, the career ban). Two chart lanes carried “No CAREER or RECORD claims”, earned by a live dry run: given a #1 debut the model wrote “Gracie Abrams’ first chart-topper”, plausible and ungrounded. The rule’s own stated premise was that “chart rows carry position, movement, peak and weeks – never career history – so a superlative about an ARTIST is always invented here”. That premise EXPIRED when #3365 started handing the same lane a grounded career block: the prompt then told the model to make no career claim and, a paragraph later, handed it one to make. Check whether a rule’s premise still holds before you treat the rule as load-bearing – a dead rule does not announce itself, it just quietly works against the surface. The fix was to make the rule CONDITIONAL on the grounding (has_career_block / career_fact), not to delete it: with no fact in hand the ban is still right. And the ban was not even doing its job. Measured live on the projection lane, the flat-ban arm wrote “a solid new entry with no chart history to lean on yet” about Dolly Parton – an act with fifty-one Billboard 200 entries. A ban on career claims does not produce silence about a career; it produces an UNGROUNDED GUESS in the other direction. Supplying the fact is what fixed it: “it’d be her 52nd entry on that chart.”
• A caveat aimed at the model becomes a hedge in the voice (#3363, owner steer). The projection framing opened “It is not a chart position, not a settled number, and not a market price” and went on to lecture that a projection stated as a finished result “is false until Billboard prints”. The owner’s steer was blunt: “don’t give any of those caveats, please, ever.” Saying what the number IS – “tracking for a #4 debut” – keeps the tense honest with no disclaimer attached, so the replacement is one clause (“say what it is TRACKING for, not what it DID”) rather than a paragraph of negations.
• A stated char band becomes the output (#commentary-tone, music drops). The owner called a shipped drop “too ai ish and too long”. The compose prompt named “a tight ~80-100 chars is the sweet spot”; the thirteen most recent drops measured 72-97 chars, dead inside that band, while the prompt’s OWN good examples ran 44-57. The model was obeying the number, not drifting. Two further causes sat in the same prompt: three of its five good examples were TWO sentences, against a rule two blocks above forbidding a second beat, and every example was all-lowercase against the persona’s sentence-case rule. The exemplars beat the rules. Fix: delete the number, make every exemplar one beat, and name the live failure shapes in place of stale ones.
• Then the same trap, in the fix (same epic). The first attempt replaced the 80-100 band with a 40-70 TARGET. Measured before/after (n=11, both models, real compose, graded blind), it made the output WORSE: the model cut the concrete half and kept the verdict, so “that falsetto hook is silky enough to…” became “Ravyn Lenae makes worry sound like a whole vibe”. A tighter number is still a number. What worked was removing the target entirely and stating the stop condition instead: you are done when you have named the part. Length then fell out on its own (84->70 chars on the prod model, 101->92 on Sonnet, over-cap lines 4/11 -> 0/11). Name the condition, not the count.
• Label the data; do not rely on a convention (#4363, scoreboard rows). The /ask scoreboard rendered Denver Nuggets vs Los Angeles Lakers, away team first. Nothing said that order. Measured on tonight’s real ESPN slates, n=6 per slate: Opus wrote 0/120 rows as “X at Y” and said the feed “doesn’t say who’s home”; both models gave 0/72 finals a home or road word. With Denver Nuggets (away) vs Los Angeles Lakers (home) it was 120/120 and 72/72 on both models, 0 flips. An @ row (Spain 2 @ Korea 1) was tried first: the Haiku commentary gate read it as an inverted score, so the dash shape stayed and the labels carry the side. The fix is in the data line, not a new prompt rule.
• Do not ask a text model to describe audio (music drops, 2026-10-18). The owner said the drops “sound like ai” a month after the fix above. The fix had asked for “something a listener could actually HEAR (… what the voice is doing)”. The model has not heard the record, so it described the sound in written copy: “that twangy string loop … is still the strangest beat to ever run a club”, “that slow, whispered groove”. The fix’s own GOOD example also became a magnet: it was a Doechii “Anxiety” line, and the bot re-dropped Doechii “Anxiety” as a paraphrase of it. The reword asks for the one REASON the track is on and says the model has not heard it. Measured n=12 per arm, real compose: 6/12 sound-portrait lines before, 0/12 after, 0/12 on a repeat. An example that names a real item is a pick, not only a format. It pulls the model to that item.

• A fix measured on one model, shipped to another (music drops, 2026-10-19). The owner said the drops “still” sounded like ai: “Megan took ‘Body’ to the Victoria’s Secret stage, so it’s coming straight to the aux” should have stopped at “stage”. The guild had moved the drop to gpt-5.6-terra, and the fix above was measured on Sonnet only. On GPT, every shipped drop ended with a clause that says she plays the track. The real compose on gpt-5.6-terra, today’s chart, release and Perplexity inputs, the guild’s real recent drops, n=12 per arm, graded by hand (“tail” = says she plays it, or that the hook is reason to; a swapped-in verdict counts too):

  arm	change	tailed lines
  base	none	11/12
  A	“SAY WHY IT’S ON … the reason you put this track on” -> “SAY ONE THING”	about 7/12
  C	ADD a named “selection tail” ban	about 4/12 tails, and verdicts took their place
  D	A, plus “hook OR opinion, not both”, plus the Travis example stops at its fact	7/12
  E	D, plus no “~20 words” in the drop’s length line	3/12
  F (shipped)	E, plus FRAMING no longer says “You control the aux”	1/12; 3/12 on a repeat

  On Sonnet the same F went from 8/12 to 2/12. Three lessons. Re-measure on the model the guild ships. The added ban moved the words, not the shape: with C the model cut “aux” and wrote “pop girls come collect” in the same place. A word count lives where you do not look: the drop prompt said “there is no target length”, but its FORMAT line pulled in the shared _POST_LENGTH with “~20 words”, and a ten-word hook left room to fill. F’s lines read as plain release facts (“KATSEYE dropped ‘Unloveu’ Friday.”); that is what the owner asked for.

• The opinion half of “hook OR opinion” (music drops, 2026-10-20). The owner called ““Janice STFU” is a mean little record.” a bad caption. It went to X with no artist and no fact, and the self-gate passed it at 0.62. The F prompt above allowed it: “a real hook … OR your plain opinion of it”, “If you have no hook, say you like it and stop”, and a GOOD example that only grades (“That SZA hook on The Weekend is peak petty”). Nothing said to name the artist. The real compose on gpt-5.6-terra, today’s Perplexity, Grok, chart and release inputs, the guild’s real recent drops, a quiet room, n=12 per arm, graded by hand (“grade” = the line rates the track and states no fact; “chart” = the line quotes the prompt’s chart list as its hook):

  arm	change	grade	chart	clean
  base	F	9/12	0/12	3/12
  A	the line names the artist and ONE confirmed hook; no hook -> pick a track you have one for; the SZA example cut	3/12	4/12	5/12
  B	A, plus the chart block says a song’s place on it “is a reason to pick it, not a hook to post”	2/12	2/12	8/12; 8/12 on a repeat
  C (shipped)	B, plus FACTS no longer says “just react to the track” when a claim does not verify (Codex on #4649): drop the claim and use a confirmed hook, or pick another track	2/12; 0/12 on a repeat	1/12; 0/12	6/12; 12/12 on a repeat

  Two lessons. A cut lane sends the model to the nearest material. With the opinion lane gone, the model used the chart block, which calls itself the “GROUNDED source”, as its hook: “Leon Thomas has ‘MUTT’ on the hot-songs list right now”. The fix was one sentence in that block, about what the block is for. C’s first run had two selection tails (“run that”); its repeat had none, so C is B within the noise of n=12. A rule the prompt states can still need a check in code. Every dry-run line named the artist, before and after, but 5 of 25 real drops in Axiom had not (Destiny’s Child, Lady Gaga, Dominic Fike, Jay-Z, Drake). The dry run’s quiet room is not the live feed. So output_checks.missing_artist refuses those lines (ship_guard.drop_missing_artist). It flags all 5 and none of the 48 dry-run lines, which include the short forms “Olivia”, “Chappell” and “MK”.

• Measure the FIXTURE too, not only the change (same epic). The dry-run room blob said “this shift is dragging”, and the after-arm kept tacking on “, perfect for a slow shift”. That read as the fix failing. With a neutral room the same prompt dropped it from 5/5 lines to 2/5, and both survivors added information rather than grading. The cue was in the harness. A before/after where only one shape resists is a reason to check what the fixture is feeding the model.

• A guard that named the words back still handed over the number (chart fall lanes, 2026-09-18). The two desk fall framings forbade every decline verb and then stated both positions (“it sits at #10 this week and was #9 last week, a HIGHER place”) and closed “The reader has both numbers and can see it.” The take did what the sentence said: “sits at No. 10 on this week’s Billboard Global 200, after ranking No. 9 last week” shipped with no decline verb, and the judge scored it 0.85 praising the clause as correct. Owner: “Don’t narrate previous position if it was higher. Just say current position.” The fix cut the number from every model-facing line on a fall (the story clause’s “(was #9)”, the watch blob’s “It was #4 last period.”, the framing’s second position and its closing sentence) and said what to do with the older position instead: leave it out. Then the first dry run found the same shape one field over. With the previous rank gone, 10 of 10 takes wrote the PEAK (“55 weeks deep after peaking at #2”, “off a #2 peak”) – because the story line still handed it over. An earlier, higher position is the steer’s shape whatever the column is called, so the peak left the fall story line too (the run stays: a count, not a position). Re-measured, n=6 on the real row: 0/6, gates 0.88-0.92. A rule that bans a phrasing does not remove a fact the prompt still supplies; the model states what it is handed. The deterministic backstop (has_prior_position) takes the take’s own rank and refuses any other position.
• The same steer, one surface over: the market number (market drop, 2026-09-18). “Miley Cyrus’s ‘Bass Persuades’ first-week forecast has dropped to roughly 61.5K equivalent units.” shipped to X at a 0.9 gate score. Owner: “Never say dropped don’t narrate negative changes just say is at.” The blob said “the market forecasts ~61.5K” and nothing else, so where did “dropped” come from? Two places the prompt itself supplied: the self-dedup block handed the model its own line from three days earlier (“forecast at ~62.4K”), and the SIGNAL rule said “report where it NOW stands, with the direction at most as color”. Thirty days of the surface’s posts showed the leaderboard angle doing the same off the blob’s “down 8 pts 24h” line, with the hint modelling it (“Y cratered 54pts to 34%”): 13 declines narrated. So the cuts, before any checker: the blob no longer carries a downward move at all (an upward one still rides), “with the direction at most as color” became “(‘X is at 34%’); a rise may ride as color”, the hint’s example became “Y is at 34%”, the forecast hint gained “is at ~61.5K units” as its model line. The rule was NOT written as a list of banned verbs (the #1140 lesson: an enumerated list summons what it names); the deterministic backstop (has_market_decline) carries the list instead, where naming the words is the point. Then the dry run on the real blob + history (n=5 per model) showed the reword was not the fix. With the earlier figure still in the history block, Sonnet wrote the decline 3/5 (“is down to ~61.5K”, “down from the ~65.2K mark”) and the guild’s GPT tier 2/5; on the Emmy board GPT went 5/5 -> 0/5 once the blob stopped carrying “down 8 pts 24h”, but Sonnet wrote “down from 62%” off the history line. Every remaining decline was written off a figure the prompt handed over. So the figures left the history block (_withhold_figures), and the leak went to 0/6 on both models – and both models then returned EMPTY 6/6, on both shapes, because the block’s other job was “return EMPTY if the numbers haven’t moved” and it could no longer see the numbers. A block that must carry a fact for one job and must not carry it for another is two blocks: the “did it move” judgment is now its own Haiku call (market_history_news, 15/15 on the live cases) that reads the figures, and the compose reads the lines with every figure withheld and a block that no longer asks it to judge anything. When a cut breaks a second job the same text was doing, split the readers; do not put the fact back.
• Four invitations to a read, and a lane hint that modelled the tail (newsroom market lane, 2026-09-16). The Drake / Polymarket card shipped “…, a pretty cold read this late.” The owner: stop at Polymarket, the tail reads as AI. No rule said to add a read; four pieces of text asked for one – “your read on it” (three times across the compose and its context), a user turn that closed “Drop the read.”, a length rule that allowed “one short clause of context” (an opinion is context), and a lane hint whose example modelled the shape (“what the number means (‘the market’s got X on pace for a #1, calling it about 62% likely’)”). Sonnet copied the example’s opener verbatim on 5/5 runs. Every fix is a cut or a reword; the hint now names the stop condition (“END the line at its anchor”) in place of the example. Then the rewords were not enough on their own, and the ablation found where the rest lived. With the prompt fixed, gpt-5.6-terra went 2/5 to 0/5, but Sonnet still folded a BACKGROUND fact in after the anchor 4/5. The first suspect, the background header’s own “fold that one in as a SINGLE short trailing clause”, was reworded for the lane and moved nothing (4/5). The length rule’s clause allowance took it to 2/5. Blanking EITHER remaining context block (the crowd read, or the background) took it to 0/5 – so the crowd block, which asks for “SENTIMENT + context to sharpen the read”, is withheld on a lane whose line ends at its anchor, and the background stays for fact-checking. Final: 0/5 on all three models, n=5 each. Ablate one block at a time and keep the one that carries facts; the first block you blame is usually not the one that moves the number (the same lesson as #858).

• One example taught a wrong FACT (newsroom certification lane, 2026-09-15). The Jennie RUBY card heroed “1x” over a take that said RIAA Gold. Nothing in the prompt told the classifier to hero a bare multiplier. Its one certification example did: "figure": "7x" for a 7x Platinum wire. The model copied the shape onto every rung, and the rungs that carry no multiplier took a wrong award — “1x” is 1,000,000 units on the RIAA ladder and Gold is 500,000. Measured over the 30 days before the report, 217 of 283 certification classifications answered a bare “Nx”, and 54 of those sat on Gold, Diamond or Silver. The fix added no rule. The example now reads "figure": "7x Platinum", and a second example shows the failing shape (a Gold, "figure": "Gold"). Measured on the real classifier over seven live wires, n=4: 24 of 24 bare multipliers before, 24 of 24 award names after. An example is a fact claim, not only a format. Check each one against the domain, not only against the schema.

• Memory over-compaction (#854/#855). Opus wrote half-length notes. The cause was reductive verbs (“Compact”/”Synthesize”/”Drop”) it followed literally — not a missing “be thorough” rule. Fix: remove the verbs and reframe, no additions. Result: Opus length matched Sonnet (≈6k→13k chars), no rule added. Adding “be exhaustive” first was scar tissue that treated the symptom.
• Recall reframe, then over-correction (#855). Telling the writer the note is searched (“write it to be FOUND, matched by meaning + keyword”) was a good, kept change. But it arrived bundled with maximalism — “the complete record”, “the full searchable record”, “breadth so any question has a hit, depth so the hit is complete”, “the small as much as the big”. The instructive part is what the rigorous dry run found (prod daily #1444, the aggressive vs softened prose, n=5 each, fixed 75-moment checklist): the two versions are a dead wash on every metric — length (13.2k vs 14.1k chars, overlapping ranges), and searchability (recall 0.343 vs 0.344, identical sd). A first n=2 pass had looked like the softened version was ~30% shorter and deeper; n=5 erased that — it was sampling noise. The real lessons: (a) Opus normalizes length regardless of “be exhaustive” wording (it wrote ~13–14k chars either way), so that maximalism wasn’t even buying the length it cost on paper; (b) the maximalism didn’t improve recall at all; (c) trust n≈5 over n=2 — a clean two-sample story is often noise. So cutting the maximalism is justified on prose hygiene (it was thrice-repeated, redundant — “complete record” and “full searchable record” and “the small as much as the big”), not on a measured length/search win. Don’t oversell a metric that isn’t there.
• Ask over-length roast (#858 / #856 — the canonical example, see docs/opus_length_drift.md). Opus ran long on thread-continuation roasts via a defend-then-soften reflex. Two things make this the textbook case:
  • The obvious cut is usually not the lever — ablate to find the real one. The instinct was to blame _OPUS_REGULARS (the don’t-punch-down block) and cut it. The ablation said otherwise: cutting it made the symptom worse (bf_hater 3.0 → 3.3) and wasn’t free (it’s the lane/villain guard). The real driver was a length-sanctioning roast rule a layer up (“a full takedown … is the fun”).
  • Fix the cause’s LAYER; don’t stack a counter-rule in a different layer. The roast sanction lives in Layer 1 (constitution.py CALIBRATION, Sonnet-era, never Opus-optimized); the length rule lives in Layer 3 (_OPUS_LENGTH). Adding a length clause to Layer 3 would just contradict Layer 1 one layer down — two rules fighting. The fix is to reword the Layer 1 line so a roast lands sharp in a line or two and drops the defend-then-soften reflex. (The same investigation surfaced Layer-1↔Layer-3 duplication — STAY IN YOUR LANE in both, DATA INTEGRITY’s 9 bullets vs the lean _OPUS_GROUNDING — i.e. more text to consolidate, not add.)
• Discourse: measure the model on the REBUILT prompt, not the legacy one (#876 / #895, full arc in docs/opus_discourse_eval.md). The first measurement compared Opus on the Sonnet-tuned 43k prompt against Sonnet, found Opus worse (a same-topic head-to-head ~6-2), and nearly concluded “keep Sonnet, don’t build.” That measured Opus on the wrong prompt — the exact confound this epic exists to fix. Given a lean discourse core, Opus flips to the strongest of the three (open-question 1/8 → 8/8, flag-planted 10/16 → 15/16, open-loop Opus 3/4 vs Sonnet 0/4 on the real surface). Four sub-traps it hit, each generalizable:
  • A pairwise “which is better” judge is high-variance — prefer per-item metrics. The same-topic head-to-head swung 6-2 → 4-4 → 2-6 on identical conditions; the stable per-post metrics (quality / open-question / flag-planted) were the trustworthy signal. Distinct from the judge-ERROR lesson below: this is judge VARIANCE — a comparative “A vs B” judge is noisier than an absolute per-item one, so never hang a conclusion on one head-to-head.
  • A reused component carries its HOME surface’s framing. Pulling the ask blocks _OPUS_CITE / _OPUS_TOOLDISC into discourse leaked meta-commentary into a room post — their “the user” / “your answer” 1:1 framing reads wrong in a broadcast post (A/B META-leak 2/8 → 0/8 once swapped for a discourse-framed line). Reuse only the surface-NEUTRAL component (_OPUS_REGULARS); reframe the rest for the surface.
  • Dry-run the REAL wired path, including its fallbacks. An offline A/B (direct _call) looked clean, but the live discourse() path has a forced-search retry (thinking OFF) the offline harness never exercised — it degraded the lean output (~4.0 sentences + selection-narration meta-leak vs the primary’s ~2.3). Skipping that retry on the lean path was the fix. Grade the actual pipeline, not the prompt in isolation.
  • Vivid prompt metaphors get parroted (instruction-echo). “plant YOUR flag first” surfaced verbatim in ~1-2/12 posts (“I’m planting my flag: …”). Reworded to “lead with where YOU land” — same instruction, echo 1/12 → 0/12, keystone metrics held (re-validated because the TASK block is load-bearing). The cause was the prompt’s own memorable noun, not a missing “don’t narrate yourself” rule.
• Two “I am the ground truth” blocks in one prompt (newsroom chart lane, 2026-09-08, #3091). The LIVE CHART block says the settled reading “SETTLES the claim”; the Perplexity BACKGROUND block under it (its header is the search surface’s framing) says its figures “win” and to use them VERBATIM. Handed both, the compose led with the research’s figure 5/5 on the real Choosin’ Texas inputs and the self-gate dropped all
  1. The first fix tried was a counter-instruction on the background (“take nothing from here as a second clause”) – it made it WORSE, 5/5 EMPTY: the model would not reconcile the two week counts and bailed. The fix that worked was structural, the #2774 rule: withhold the research block once a live reading has settled the claim (5/5 ship). A reused block carries its home surface’s authority claim; when two blocks claim it, drop one rather than referee between them.
• Two dead ends on the reply surface’s verbal tics (2026-08-03, after the #1970 payload-list cut). The cut halved the pundit frame (realnoun 28% → 11%), and the owner could still search Discord and see the residue — 21 hits for “flex”, a wall of “the real X” and “nobody’s talking about”. Two more levers were tried on the same rig (Opus, 18 real targets, production context, 2 runs/arm). Both failed, and the failures are the useful part:
  • Showing a surface its OWN recent output makes the tic WORSE. Feeding the last 8 drafts in as “YOU POSTED THESE RECENTLY (vary it — don’t reuse the same opener, verb, or sentence shape)” — the block the wire desks and market drops already carry — took any-tic 9/35 → 19/36 and register flags 1/35 → 6/36. The block is a primer: the phrases you show a model are the phrases it reaches for, and a “don’t reuse” instruction does not cancel that. This is the epic’s core lesson in a new costume, and the trap is that the block is proven elsewhere — the desks use it against topic repeats (did I already post this story), a different axis from phrase repeats. Reusing a component across axes needs its own measurement.
  • A few-shot pool’s sentence frames were a wash. 5 of the 17 CALIBRATION exemplars are the appositive shape (“That ‘See you at the EMMYS’ is the cleanest little jab”), so a k=4 sample carries ~1.2 of them every call — a clean-looking cause. Cutting the block entirely (realnoun 2/36, any-tic 9/36) and sampling only non-appositive lines (5/35, 9/35) both land on baseline (4/36, 7/36). The pool teaches register, not frame; #1957’s “exemplars are a wash” result extends to this axis too. What did work, once the owner pushed back a third time: NAME the phrase. A deterministic detector (utils.x_reply_draft.tic_hits, calibrated on both corpora – it trips 39% of a live week’s drafts vs 6% of the owner’s own posted replies) plus ONE re-ask that hands the model the exact words to drop took tics 9/36 → 2/36 (25% → 6%, p=0.046, register a wash) – i.e. down to the owner’s own rate. The pair is the lesson: “vary it, here’s everything you said” PRIMES the habit; “you wrote ‘the real X’, say it without that” removes it. Same axis, same surface, opposite results, and the difference is specificity. The retry is taken ONLY when it comes back clean, so the guard can never degrade a good line – a fix that can only help is worth shipping at a lower evidence bar than one that trades.

Evaluating model effectiveness (how to measure, not just what to change)

Half of this epic’s lessons were about evaluation method, because a prompt change is only as trustworthy as the measurement that justified it. What we learned:

1. Pick the metric and the axis before you edit. “Better” is meaningless. Name the axis — searchability, length, depth-of-record, fabrication rate, adherence — and define how you’ll measure each before touching text, so you can’t rationalize a change after the fact.
2. Measure with the REAL pipeline, not a proxy. Recall means the production path: max(prose_vector, tags_vector) cosine over the real query set against the real embedding model — because that’s what actually retrieves the note. Eyeballing the prose for “searchability” measures nothing.
3. n=2 is noise; trust n≈5. A two-sample dry run told a clean story (softened was “~30% shorter and deeper”); n=5 erased it (length a wash, depth inconclusive). Model output is high-variance — a clean small-n result is more likely sampling luck than signal. Run enough samples to see the spread, and report the spread (min–max + sd), not just the mean. A mean without a range hides whether the difference is real.
4. Validate the metric itself — a broken metric is worse than none. The n=5 depth judge returned coverage counts above the fixed checklist size (depth

   1.0), i.e. it was nonsense, and a careless read would have “concluded” from it. Sanity-bound every derived metric (covered ≤ total) and discard it loudly when it violates the bound, rather than quoting a number that can’t be real.

5. Isolate the parameter you’re testing. Hold everything else fixed (same source data, same queries, same fixed checklist as the depth denominator) so the only moving part is the one prompt change. A floating denominator (the judge re-deriving the moment count per call) injects variance that swamps the signal.
6. Ablate cut vs add vs reword separately. Don’t test a bundle. #858’s table (baseline / cut-the-block / add-a-clause) is the model: it showed the obvious suspect (cut the regulars block) made it worse and the real lever was elsewhere. You cannot tell which token did the work unless you move one at a time.
7. A wash is a result. If the metric says no difference, say so and decide on other grounds (prose hygiene, owner taste) — don’t manufacture a metric win to justify a change you already made. Honest “no measurable effect” is more useful to the next session than an oversold number.
8. Dry-run on real production data. Pull live notes/logs (/debug/query, Railway EVENT logs); synthetic fixtures don’t carry the live distribution that produces the failure.
9. A judge is itself a model — validate a FAIL before acting on it. The eval judge can be wrong, including failing a correct answer. Worked example (#893): the hardened recap eval failed an Opus recap for saying “SGA won the 2025-26 MVP” — and I first read that as an Opus weakness (“over-asserts an outside fact”). The statement is true; the recap had correctly grounded a real recent fact about the exact topic the room was debating, which is what recap is supposed to do. The judge erred — it couldn’t web-verify a recent result and treated “couldn’t verify” as fabrication, against its own rubric. Two compounding traps: (a) acting on a single judge verdict as ground truth, and (b) it nearly produced scar tissue — a “lean recap core” rebuild for a weakness that does not exist (the #861 wash STANDS). The bar a fabrication judge must enforce is “contradicted by search → FAIL,” never “couldn’t verify → FAIL.” Before you act on a judge FAIL, check the judge was right.

Eval fidelity: model production, or you measure nothing

An eval is only worth its verdict if it feeds the model what production feeds it and exercises the path production runs. Three failure modes this epic closed (#886, the ask/memory eval lift):

1. Feed the production-shaped context, not a barer frame. Every real /ask supplies channel_descriptor (“where am I”) + asker_name (“who’s asking”) + the rich message buffer (reactions / reply-graph / author-tags). The synthetic cases passed none of it, so the model saw a thinner frame than any live call and the verdict measured a different distribution. Fix: feed the same context the cog feeds (cogs/ask.py), verbatim shape.
2. Wire the REAL tools/pipeline — never a reimplementation. eval_ask_tools now builds the real SportsDataHub (ApiSportsProvider + SgoProvider) and calls the real format_scoreboard; the thin handlers just call hub.live_games()/upcoming_games()/recent_finals() exactly as the cog does. A handler that reimplements cog logic in the eval drifts from prod silently — then it grades a fiction. Reuse the production function; the eval’s job is routing + answer quality, not a parallel implementation.
3. Grade the model actually SHIPPED per surface — read it from the DB (cost discipline). Don’t grade every model every run. The scheduled suite reads each surface’s enabled model from the per-guild /menu Models override (settings KV) via resolve_eval_models/merge_enabled: flagless → the single enabled model (the surface default UNION any guild override), so it pays 2x ONLY when a guild genuinely split the surface across models. Wiring detail worth remembering: GitHub’s runners cannot reach Railway’s internal DATABASE_URL host, so the read uses DATABASE_PUBLIC_URL (the public proxy) and is fully fail-open — no URL / unreachable → fall back to the surface default, never block the suite on a DB hiccup.
4. Pin the production model in an ad-hoc dry run too, not only in the suite. Item 3 makes the scheduled suite grade the shipped model. A one-off A/B you run by hand needs the same discipline, and it is easy to skip because the default client is right there. Worked example (#2148): the music desk’s watch lane claimed a chart RE-ENTRY it could not prove, and the fix cut the phrase that offered the claim. The first A/B ran on the surface’s DEFAULT model and scored 0/6 on the BEFORE arm — the bad wording produced no bad output, so the result justified nothing and the edit looked unnecessary. Re-run pinned to claude-opus-4-8, the model that composed the live post, the BEFORE arm scored 3/5 and the AFTER arm 0/5. This is the same Opus-vs-Sonnet split the top of this file describes: a phrase that is inert on one model is a live instruction on another. So read the surface’s enabled model first, pin it, and only then believe the numbers. An unpinned A/B measures a model nobody ships.

Validate the gap is real before you build the fix (proportionality)

The fix is usually smaller than the issue makes it sound — measure the gap before building machinery for it. Worked example (#892): the issue asked for a live-embedding behavioral eval of chunked recall, and I started building it. On inspection the gap had mostly already been closed: the chunk mechanism (the two-stage rank_by_similarity rerank, base64-f32 encode/decode) was already deterministically unit-tested in tests/test_embeddings.py and gating CI, and a live-embedding eval would have been dormant anyway (evals.yml has no OPENAI_API_KEY, so embedding cases self-skip). The one genuinely untested seam was the cog wiring — that cogs/ask.py:_vector_recall passes chunk_rerank_top through. The proportional fix was a ~10-line deterministic wiring test, not a live eval harness. Pattern: enumerate what already covers the risk (unit tests, an existing eval, a dormant-in-CI gate) before adding more — sunk-cost on a half-built heavier fix is not a reason to ship it. (Pairs with the CLAUDE.md “proportional to the problem” rule and the #567 resume-spine → 10-line cog_unload example.)

The standing directive

Apply this to every piece of text Opus sees — constitution, persona, per-surface prompts, tool descriptions, context-block labels, eval judges. Each is a lever to be optimized by cutting and rewording for Opus, not a place to accrete rules. When you’re tempted to add: stop, find the cause, cut it instead.

See also

• docs/opus_length_drift.md (#856/#858) — the worked investigation behind the ask-roast example: the full ablation table, the 4-layer prompt assembly, the cross-layer tension, and verbatim before/after rewords of the Layer-1 roast rule.
• Epic #849 (per-model prompt + eval rebuild) and its milestones: ask #859, memory #860, recap #861, discourse #862, other tiers #863, eval coverage #886.
• The eval-fidelity arc: #886 (coverage lift), #887 (DB-read model axis), #890 (production context + real sports tools), #892 (the proportionality correction), #893 (the judge-calibration correction).

Worked example: the compose history is cut to the rows its job needs (2026-11-03, #4926)

The wire composes (market drop, the music desk and newsroom drop and record lanes, the pop, sports and cinema desks) put a “YOU POSTED THESE RECENTLY” block in the prompt. They built it from _sched_recent_history, which returns the 180 rows the dedup GATES need. So every compose carried about 16k characters of history: measured on gpt-5.6-terra, 12,940 input tokens with 180 rows and 8,970 with 30 (-31%).

The block has one job: stop a reused opener, verb or sentence shape. The subject repeat is the gates’ job, and they still read 180 rows. So the metric was the repeat rate, measured on the production model against the FULL 180-row history (scripts/dryrun_compose_history.py, 6 real snapshots x 2 samples):

case (gpt-5.6-terra)	arm	opener repeat	shape repeat	judge pass
market drop	180 rows	2/60	1/60	60/60
market drop	30 shared	0/60	2/60	60/60
market drop	30 shared + own	0/60	1/60	60/60
music record	180 rows	10/36	0/36	-
music record	30 shared	11/36	0/36	-
music record	30 shared + own	12/36	0/36	-
sports take	180 rows	0/24	0/24	-
sports take	30 shared	0/24	0/24	-
sports take	30 shared + own	0/24	0/24	-
pop take	180 rows	0/24	0/24	-
pop take	30 shared	1/24	0/24	-
pop take	30 shared + own	0/24	0/24	-

Every opener hit starts with the subject’s name or title (“Dominic Fike’s ‘Babydoll’…”): the cases come from real posts that are still in the history, so it is the same subject, not a reused template. The desk cases put the desk’s own real takes at ranks 40-160, where a low-volume desk’s posts sit in the shared read. Those rows are past row 30, which is why the shipped cap also keeps up to 10 of the calling desk’s own rows (an acceptance-review fix). The result is a wash on quality, so the cut is pure cost: the block was bloat-without-payoff (method step 6). utils/wire_lanes.py:read_compose_history holds the cap; the gates do not call it.

Worked example: the awards write-up (2026-08-30)

The owner asked for two things at once — “cut wording down, it’s too long and repetitive, we repeat the all time context too much”. Both were caused by existing text and existing DATA, not by a missing rule, so both fixes were cuts.

• Length. The weekly intro prompt carried the maximalism this file warns about: “give it the FULL fanfare, the same energy as the big monthly ceremony … make the one crown land big”, plus “Two or three sentences”. Sonnet obeyed the maximalism, not the count — it wrote 3.4 sentences. Cutting the two fanfare clauses and asking for “One or two sentences” landed it at 2.0. The hype prompt also said “2-3 sentences” twice and gave the same instruction twice (“make them feel genuinely SEEN” … “specific enough they’d feel genuinely seen”); cutting the duplicate and halving the count took the pair from 717 to 535 chars (n=5, real guild data, pinned to the shipped model — Sonnet on both surfaces).
• The repeated “all time” context was a DATA bug, not a prompt bug. _winner_receipts fed the winner’s top_post_text — their all-time most-reacted message — into a prompt that says “THIS period”. The model was doing as told with what it was given: 5/5 hype-ups quoted a months-old graduation post as the week’s moment. No rule would have fixed that. Diffing the seed’s highlight reel against the window-start snapshot gives the real this-period post, and the leak went to 0/5. Pattern: when a model keeps reaching for the wrong period, check what period the DATA is from before you write a rule about it.
• Measure the claim, not the vibe. The accuracy axis was named first (“does the post assert a rank the winner did not take”), detected with a regex over the real output, and run on the real board shape that produced it. It caught what a read-through would have excused as hype: “Player of the Week, no debate” and “and it wasn’t close” — 4/5 on a winner who placed 4th.

Worked example: the RIAA board takes + scar removal (2026-09-06)

The owner’s report was a style one – “we’ve been writing like AI lately instead of just reporting plainly” – against one shipped card: “2026 has been kind to Future’s catalog: 59 Platinum certifications so far this year, 9 clear of Kanye West’s 50 among today’s top acts, with SZA’s 27 not far behind.” Every part of that sentence traces to a line in utils/riaa_boards.py:compose_inputs. No rule was added. Four lessons, three of them about the SAME mechanism.

• The take reuses the framing’s vocabulary, so every word in the prompt can ship. The runner-up line read “The runner-ups are giants too”. Measured over the 120 posts @tootsiesbar shipped in the 48 hours to 2026-09-06 16:31Z, three said “giants” back, and the word is in no board’s data. Model-facing text is not only instruction; it is also a word list.
• The first rewrite was WORSE, and only a second dry run caught it. Cutting the adjective looked like the whole job. The replacement – “Name one or two runner-ups with the tally the block gives each, not only the gap behind the leader” – did remove the tells and quietly tripled a different failure. On the Platinum-singles board, whose block states exactly ONE gap (the leader against second), the take began working out a SECOND one: “35 clear of Future (102) and 53 ahead of Lil Wayne (84)”, where 53 is the model subtracting two rows. ungrounded_numbers drops that before the self-gate, so the cost is a spent compose. Invented-number rate, n=8 per wording on the same board: the original 2/8, the first rewrite 6/8.
• The obvious cause was the wrong one – ablate, do not reason. The hypothesis was the word “gap”: the instruction said it, so the take wrote gaps. A wording with no “gap” in it at all (“Name one or two runner-ups by the number the block prints beside each”) came back 5/8 – no better. The real work in the old line was done by “the runner-ups are GIANTS TOO”, which makes each runner-up its OWN subject. Without that, “name them” is satisfied most cheaply by extending the comparison already in the sentence, so the model writes a second gap. The wording that keeps the frame and drops the adjective – “Name one or two runner-ups. Each stands on its own number from the block, so state that number and nothing more” – lands at 1/8, better than the original, with 0/8 tells. Same lesson as #858: the thing you are about to cut is usually not the lever.
• Deleting the line scored best and was still the wrong call. A fourth arm removed the runner-up line outright. It is the cleanest on every failure axis – 0/10 invented numbers, 10/10 through the self-gate on the fastest board against 8/10 with the line, and 0/10 of the identity claims in #3019 against a 1/10 baseline. It also stops doing the line’s job: all ten year-board takes named the leader and second and nobody else, so a ten-row board came out as a duel. “Cut it” is the default this file argues for, and the test of a cut is whether the text was buying anything. This one was.
• Use the echo deliberately when a downstream guard bans a word. The certified-units opening said the board “adds” the two formats. The take answered “combined”, which output_checks.has_row_arithmetic reads as a sum of the drawn rows and drops before the self-gate (live 2026-09-05 17:06:34Z, one compose spent; the 18:03Z retry shipped only because it happened to write “spanning”). Rewording the verb to “spans” did NOT fix it: n=5, 5 of 5 still said “combined”. Putting the word SPANNING in the instruction did: 5 of 5 said “spanning”, 0 of 5 said “combined”. Seeding the word you want beats removing the word you don’t, and it beats banning it.
• An opening a subject cannot carry becomes a personification. The year board was told to “Open on the YEAR and the run of new plaques it has produced”. A year produces nothing, so the only way to follow that instruction is to give the year a personality. That is where “2026 has been kind to Future’s catalog” came from. The instruction now names the count.
• A scope instruction outlives the scope it describes. The same block ordered the take to keep “among today’s top acts”. That was true while the board ranked a 40-name roster and false since #2906. All five before-takes carried the false fence; none of the five after-takes does. When a surface’s population changes, grep the prompt for the old population – the fence lives in the model-facing text as well as in the code.

Method note: the axis was named before the edit (does the take carry an AI tell, and does it carry a fence the board does not have), it was counted over the REAL shipped corpus first, and the before/after ran on the real boards through the real wired compose path (compose_market_drop with the desk’s own _CONTEXT), scored by the real self-gate.

And one axis was NOT named before the edit, which is why the first rewrite shipped a regression into a PR. The n=5 pass measured the tells and the false fence – the things the owner reported – and nothing else. The invented gap only appeared when the second pass widened to every board and counted every guard. A prompt edit can move an axis you were not watching, so measure the guards the surface already runs, not only the axis you set out to fix.

Count the failure directly, not through the guard that happens to catch it. The invented gap was found through ungrounded_numbers, which flags a figure the source never states. That guard undercounts this failure: on the 2026 board the take wrote “28 ahead of Rod Wave (31)”, a computed gap whose value happens to equal another act’s row count, so it reads grounded and ships. The honest metric was a direct one – any “N clear of / ahead of” the block does not itself state.

The scar in the same file

The owner asked for the scar to come out in the same pass. Four kinds were there, and naming the kinds matters more than the four cuts, because each recurs:

• A rule with two definitions. Four boards ended their own scope with a row-arithmetic ban and the shared tail banned it again in the very next sentence. One definition now, in the tail.
• Our changelog, inside the prompt. The Diamond scope carried “That was NOT true while the board ranked the streaming roster (#2919), and it is true now (#2915).” A model cannot act on our PR history. It was the only issue number in any model-facing string in the file.
• A proof that was another board’s data. The same sentence proved “a catalogue act IS on this board” with “Garth Brooks leads the album board with 10” – on the SINGLES framing. _number_grounding reads the framing, so a singles take quoting 10 would have passed the guard. The claim survives; the proof is gone.
• Engineering detail the take can never state. The year scope explained our credit parser with two examples of a credit we cannot split (“‘Of Mice & Men’ is one band, ‘Parmalee & Blanco Brown’ is two”). The residual is still stated; the worked examples are in the code, where the next reader needs them.

The head also said one thing twice, twice: “do NOT list the rows” beside “do not recite the leaderboard”, and the card’s credit stated both before and after the rule that depends on it.

Worked example: ablating the RIAA framing line by line (2026-09-06)

Owner steer: “deleting prompt lines is the best strategy, inspect more and ship”. So every line of the RIAA board framing was deleted in turn and measured against its own baseline – five candidates, three boards, n=10 per arm, real model, real wired path, real self-gate.

The steer is right about ONE line and wrong about the other four, and the shape of the result is the useful part.

deletion	platinum invented gap	year identity claim	acts named
(baseline)	1/10	0/10	3.0 / 2.6
the runner-up line	0/10	0/10	1.1 / 2.0
“Never invent a figure”	4/10	3/10	3.0
the credit-tag ban	5/10	1/10	3.0
“in a line or two, then stop”	7/10	1/10	2.9
the re-COUNT clause	4/10	1/10	3.0

Four unrelated cuts all degrade the SAME board, so the lever is not any one line. A shorter framing leaves the take room to invent an angle, and the angles it reaches for are the ones the guards exist to catch: a gap the block never stated, and an identity claim. Cutting “Never invent a figure” is the clearest case – it is one short sentence, it looks like boilerplate next to a longer grounding rule that already says the same thing, and removing it quadrupled invented numbers and took identity claims from 0/10 to 3/10.

Only the runner-up line was free of that, and only because deleting it removes the comparison itself. No runner-up in the take means no gap to work out. It is the one deletion shipped, and its cost is stated rather than hidden: act coverage falls from 3.0 names to 1.1 on the Platinum board. The card still draws every row, so the board is not lost – the take just stops narrating it.

One cut was board-dependent, which is its own warning. Deleting “in a line or two, then stop” is the best result on the YEAR board (123 chars against 142, one sentence, no new failures) and the worst on the Platinum board (7/10 invented). A cut measured on one board and shipped everywhere would have looked like a clean win. It appears in seven files across the repo; none of them were touched.

Method note. output_checks.count_sentences reads a decimal point as a sentence end, so any take carrying “336.5M” measures as four sentences. Every length reading here uses characters instead. A metric that is wrong in the same direction on every arm still ranks the arms correctly, but it cannot be reported as a sentence count, and the production answer_length telemetry had the same artifact. #3022 fixed it: count_sentences now uses output_checks.split_sentences, which does not end a sentence at a decimal or at “No. 1”. answer_length.sentences before that fix reads high for any reply with a figure.

Worked example: the age of the record (2026-09-06)

Symptom. Two X posts off kworb chart rows closed with an age: “pulling 14.2M streams a track over a decade old” and “a catalog cut over four decades old still finding new spikes”. Axiom showed 20 shipped watch-chart takes in 30 days with the same tail (“thirteen years deep”, “sixteen years after its release”, “twenty years on”). The owner asked to cut them.

Cause, found in existing text. The blobs for those lanes carry a position, a move and a play count, and no age. The shared _CONTEXT block (cogs/music_desk.py) carries an AGE / STAGE paragraph written for the Luminate tracking head, whose line really does say “catalog, out 4.2 years”. It opened as a fact about every post:

AGE / STAGE: the head names the release’s milestone – ‘debut week’, ‘first month’, … Put that milestone in your own words, fresh each post … a catalog album’s weekly number is a this-week read (‘46M this week, eleven years deep’) …

Twenty-odd lanes hand the model that paragraph beside a row with no age in it. The model did what it was told and supplied the age from its own knowledge. The _drop_ungrounded digit check already dropped “14 years old”, which is why the shipped tails are all in words.

Fix: reword the cause into a condition, no counter-rule stacked elsewhere.

AGE / STAGE: a Luminate head names the release’s milestone – … WHEN the head states one, put that milestone in your own words … When the material states NO age or milestone – a chart position, a rank move, a play count – say nothing about how old the title is: a chart row carries no release date, so its age is not in the material.

Measured (real model, n=6 per arm, the exact production blob + framing). Bruno Mars “Locked out of Heaven” #50 -> #33 on the Global Weekly: before 3/6 takes carried an age (“pushing 14 years old”, “years past its debut”, “aging like wine”), after 2/6 (“14 years deep”, “the 2012 cut”). The Billie Jean Spotify-jump blob went 0/6 -> 0/6 on this model. So the reword lowers the rate and does not zero it, and n=6 cannot separate 2/6 from 3/6. A prompt rule the model breaks intermittently gets a deterministic backstop (output_checks.ungrounded_age_claim, reason=ungrounded_age), the same call as has_career_claim and the #2081 digit check. Measured against 30 days of shipped takes (n=5774): the pattern matches 67; the lanes whose blob states an age (tracking, lookup_board) stand down, and the rest are true fabrications.

Lesson. A paragraph that is true for ONE lane’s material reads as an instruction on every lane that shares the block. When a shared block describes the material (“the head names X”), write it as a condition on the material, not a fact about it.

Worked example: the called-shot benchmark fence (2026-09-07)

#3014 handed the called-shot compose ONE new fact — what the artist’s own last album opened at. The existing fence justified itself with a sentence that the new fact made false:

before: ... a projection row carries a rank, a unit count, a sales split and a label, and nothing else. Anything more would be you guessing.

Per the method, the stale clause was reworded, not counter-ruled — no second rule stacked on top saying “but you may also use the benchmark”. The first attempt then over-corrected, and the live dry run is what caught it:

attempt 1: ... Two points are not a trend: say what the gap IS, never that it is a rise, a fall, a record or a best-since

Three live composes on the real building chart, all gating 0.92, and one of them wrote “up big from the 18k his last album opened with” — a direct violation that the judge passed without blinking. The rule was wrong, not the model. Handing it two numbers and forbidding it to say which is larger is incoherent: comparing is saying which is bigger, and the comparison is the entire reason the figure was added.

after: ... and the ONE prior opening above. That figure is this artist's OWN last album, and comparing the two numbers is allowed -- it is the reason you have it. What you have is TWO POINTS, not a career: never 'his biggest ever', never 'his best since', and never a claim about where his career is heading

Five further live composes, all 0.92, all landing on arithmetic over the two given numbers (“more than 5x his last album’s 18K opening”) with no career or trajectory claim.

Two lessons, both already in this doc and both re-earned the hard way:

1. Ban the thing you actually mean. “Never say it rose” was a proxy for “never claim a career trajectory”, and the proxy caught the legitimate case. Opus follows literally, so an over-broad ban is either obeyed (losing the feature) or ignored (as here) — and an ignored rule is still prompt bloat.
2. A passing self-gate is not a passing fence. The judge scored the violating post 0.92 because the post was accurate and well-formed. Only reading the output against the rule showed the rule and the output disagreeing.

Worked example: the gap with nobody in it (2026-09-09)

The projected-units board (chart_boards.projected_top_sales) shipped this to X on 2026-09-08, self-gate 0.88:

Projected Billboard 200 units for Sep. 19 span ~69K to ~34.8K, with HITS Daily Double forecasting a ~4K gap at the top.

Ella Langley’s “Dandelion” was the #1 at ~69K and the post never named it (owner: “this should’ve narrated dandelion”). The cause was in the existing text, twice over. The board’s headline (THE FACT the model reads) opened with the span and named the leader second, and its steer said:

before: Then lead with the SPREAD rather than who is #1: the ranking board already said that, and a units order IS the chart order, so without its own angle this is the same post twice. The fact here is the DISTANCE between these numbers

That clause was written in #2098 to stop four projection boards opening with the same leader. It over-reached: “rather than who is #1” reads, to a literal model, as “do not name who is #1”, and the take obeyed. The reword keeps the angle and restores the subject (_PROJ_STEERS["proj_top_sales"]):

after: Then NAME THE ALBUM ON TOP and its units figure: the leader is the subject of this take, and a gap with nobody named in it is not a post. YOUR ANGLE IS THE DISTANCE -- how far clear of #2 that album is, or how tightly the rest of the board is packed. The ranking board owns positions and moves; you own the leader's units and the gaps.

The headline now opens with the leader too ("Dandelion" by Ella Langley is tracking for ~69K units, the most on the projected Billboard 200 for Sep 19, per HITS, ~4K clear of the #2 album. The top 10 runs from ~69K down to ~34.8K.). The class sweep found the same clause on the platform play-count steer (_PLATFORM_STEERS["platform_streams"]) and reworded it the same way.

Live dry run on the same HITS document (scripts/dryrun_chart_boards.py --compose):

Hits Daily Double has Ella Langley’s “Dandelion” leading the projected Billboard 200 for Sep 19 at ~69K units, only about 4K clear of Olivia Rodrigo’s ~65K, tightest race atop the chart in a minute. – gate 0.88, SHIP

The rank board’s take from the same run named Dandelion too, and the two passed the wording-only dedup the chart cards ride (SequenceMatcher 0.43 against a 0.6 threshold; 0.49 against the rank take that actually shipped on 2026-09-08). The lesson is the “ban the thing you actually mean” one again: the thing meant was “do not build the take around the leader’s rank”, and the proxy “rather than who is #1” banned the subject.

Worked example: a clause the prompt demanded and the lane could not supply (2026-09-14)

Owner steer, on three shipped @tootsiesbar posts: “cut second sentence catalogue talk”.

The takes:

Billie Eilish cleared 8 billion streams on the year, a catalog run that refuses to let go. Kanye West cleared 8.3 billion streams on the year, a massive catalog-and-current run. Tyler, The Creator cleared 3.2 billion streams on the year, another massive catalog-and-current run.

The lane is the music desk’s ANNUAL year-to-date milestone (cogs/music_desk.py, _CONTEXT MID-YEAR MILESTONE + the streams-milestone note). Four of four annual takes that night ended in a clause that restates the figure in words. The self-gate scored them 0.90 and called the filler “one clause of real context”, so nothing downstream was going to catch it.

The cause was a mandate, not a missing ban. The paragraph read '<artist> cleared <X> streams on the year' + ONE short trailing clause, then STOP, and it named three sources for that clause: the momentum trend, the year-end projection, the rank. All three are withheld on this lane – momentum is gated to the non-annual streams facts, the projection is banned by the streams-milestone note in the same context, and a rank only arrives when a chart or qualifier block lands. So on most slots the model was told to write a clause and handed no fact to write it from. It wrote one anyway.

The paragraph already carried the counter-rule. “never a closing flourish” sat two clauses after the mandate and lost to it – the canonical shape this file warns about: a ban stacked beside the instruction that causes the behaviour does not cancel it.

The fix is one reword: the clause is CONDITIONAL on a fact. The format line now stops at the figure, and a trailing clause is allowed only to carry a hard fact a context block states. Given none, the post ends at the figure. The streams-milestone note appended for a measured report lost its own half of the sanction (“one sharp human beat about how big it is”), but only on the annual lane (streams_milestone_note(annual=...)). A NON-annual measured report heads its blob with a grounded release stage – “debut week”, “first month, 3 weeks in”, “catalog, out 4.2 years” – and the shared AGE / STAGE paragraph asks the take to put that milestone in its own words. The stage sits in the BLOB, not in a context block, so the annual wording would have stripped the stage framing off every weekly report whose fail-open supplemental blocks came back empty. The first version of this PR missed that and the Codex reviewer caught it. The lesson generalizes past this fix: a rule keyed to “a fact from a context block” silently excludes the reading’s own head, and a shared block is appended on more lanes than the one you are looking at – so check which lanes it reaches before you narrow what it allows. The annual head carries “year-to-date” and no age at all, so it loses nothing. Dry-run both halves: the non-annual takes still read “four years deep and barely slowing down” / “six weeks in and still holding a real number”.

Measured on the real wired path (compose_market_drop with the desk’s own _CONTEXT and the real blob; Sonnet, the lane’s default; the three reported artists, n=5 each, run with and without a Billboard-standing block):

arm	takes	ending in an invented clause
before	18	18
after	30	0

The chart fact still rides when the block names that artist (“cleared 8.3 billion streams on the year, and ‘Graduation’ is still charting at No. 148 on the Billboard 200 this week”), and the two artists the block did NOT name correctly ignored it.

Two guards were measured, not assumed, because a shorter take could have gone dark two ways. The self-gate scores the bare figure 0.90-0.92 against the real blob (the same score the flourish got), so nothing new is dropped. The arbitrated dedup still ships four same-shape annual takes in a row: the mechanical gate matches the shape in both arms, and the judge overturns it on the artist and the number. A first dedup run said the opposite and was wrong – the harness called the judge with the wrong signature, so every judge call raised and the fail-closed direction left the mechanical verdict standing. Check that your harness’s judge actually answers before you read its verdict.

A shared block is read by more surfaces than the one you are looking at (2026-09-14)

The second Codex finding on #3252, and the more general of the two. The clause rule was written as “a HARD FACT a context block below states”. That phrasing assumes the surface HAS context blocks. cogs/music_desk.py:_CONTEXT is shared with cogs/music_alert.py, whose _fire passes it with no supplemental blocks at all – so on the alert path the rule named a source that does not exist, and the only material the take had (the market blob, carrying the line’s own move) fell outside what the rule allowed. Fixed by keying the rule to the MATERIAL rather than to a block that may not be there.

The predicted symptom did not reproduce, and saying so matters more than the fix. The review expected the alert to discard the move and post the already-known year-to-date count. Measured on the real classifier + the real compose (an annual line_move, Sonnet, n=8 per arm), the figure rode in 8/8 of every arm, the flagged wording included – the model used the blob’s pace line regardless of what the rule said it could use. The fix is right because the wording was wrong, not because it rescued a post.

What the measurement DID find is a weakness that predates the PR. An annual line_move alert fires because the market-implied number moved, and the take almost never says so:

arm	names the MOVE	names the figure
main (pre-PR control)	0/8	8/8
the flagged wording	0/8	8/8
the fix	2/8	8/8

main scoring 0/8 is the whole point of running a control: without it this reads as a regression the PR caused, and it is not one. The cause is upstream of any clause rule – the MID-YEAR FORMAT makes the year-to-date figure the mandatory lead and demotes the move to a trailing clause, on a lane whose entire reason for firing is the move. That is a separate change on a separate lane, so it was FILED, not folded into this PR.

The scar in the same block (2026-09-14)

Owner steer in the same session: “cut scars”. Two of the four kinds this file names were in cogs/music_desk.py:_CONTEXT, and both are the FIRST kind – a rule with two definitions.

• The LENGTH rule had THREE definitions across two layers, and the numbers disagreed. The system prompt owns it for every desk surface: claude_client._POST_LENGTH (“ONE sharp sentence, ~20 words, then stop … Pick the SINGLE sharpest beat and let it carry”), plus the market-drop tail (“never the figure AND a projection AND a trend AND a flourish; if the figure alone lands, stop AT the figure”). _CONTEXT then restated both – “HARD LENGTH LIMIT (the #1 rule): ONE sentence, ~25 words” and “Stacking the figure AND a projection AND a trend AND a flourish is a banned paragraph”. The second clause is near-verbatim the system tail; the word counts are simply different numbers for the same rule. Cut to the one thing this block alone knows: the several context blocks are OPTIONS for the single clause, not a checklist.
• The MID-YEAR paragraph closed on its own opening rule, plus a stale fence. “Use the line’s figure as given – not the full every-digit count, and not a rounded-off version.” The paragraph opens 60 words earlier with exactly that (“the figure EXACTLY as the line gives it … both digits kept, never rounded off”). The “every-digit count” half fences a shape the model is never handed: format_reading has put humanize_units(value_so_far) – “8B” – in the blob since the owner’s “no one needs the exact count” steer. A fence for material that no longer exists is the “scope instruction outlives its scope” case again.

Ablated on the real wired path before cutting (compose_market_drop, the desk’s own _CONTEXT + the real blob, Sonnet, n=5) across three cases, so a cut that fixes one axis could not quietly break another:

case	axis	baseline	scar cut
annual, no chart block	ends at the figure	5/5	5/5
annual, chart block names the artist	quotes the chart fact	5/5	5/5
weekly catalog album	states the release stage	5/5	5/5

Mean length moved 9.0 → 9.0, 20.8 → 22.4 and 22.6 → 19.8 words: noise in both directions at n=5, no trend. So this cut is justified on prose hygiene alone, not on a measured win – the same honesty the #855 entry insists on. 287 characters out of a 3917-character block, and one contradiction the model could have resolved either way is gone.

Two candidates were examined and NOT cut, because each still earns its place: the AGE / STAGE paragraph’s closing “a chart row carries no release date, so its age is not in the material” (it reads like engineering detail, but it is the reason the rule holds, and that lane’s whole failure mode was the model supplying an age the row never stated), and the streams-milestone note’s “no exact digit-string” (it fences a FABRICATION the model can produce unprompted, not an echo of the material, so the stale-fence argument does not apply).

Worked example: the weekly streams beat (2026-09-29)

Owner steer on weekly streams posts that ended “a steady start”: “any commentary should come from the stats, like the qualifiers. If it’s just an empty human beat, remove it.”

The #3252 fix above made the clause CONDITIONAL on a fact, but only on the annual lane. The weekly lane kept three sources of an empty beat, and all three were cut, not countered:

• streams_milestone_note said “Then one sharp human beat about how big it is.” It now has one rule for every measured report: a clause carries a hard fact the material states, else the figure is the whole post.
• The RELATIVE READ block handed the model a Songstats momentum verdict. Its STEADY branch said “a normal week (say so plainly)”, so the model said so. The block, the momentum fetch, and lookup_artist_momentum are gone.
• The blob head carried the release stage (“debut week”, “catalog, out 4.2 years”), and the shared AGE / STAGE paragraph asks the take to put a stated stage in its own words. A measured report now builds its blob with include_stage=False. Market reads keep the stage, because the paragraph is shared with them.

A rank from the background block and a verified qualifier still ride as the one clause, because they are stats.

A fourth source was cut the next day. The shared _CONTEXT block closed on its only worked clause: “and the dailies are cooling about 22% off the peak.” After the momentum block was gone, the example still modelled a streams-trend clause. The shipped Miley line (“even as Miley’s Spotify daily streams cool”, 2026-09-29) is that shape with the figure dropped. The example is cut; the no-fact example (“sza cleared 6.2 billion streams on the year.”) stays. The owner also cut “a trend figure” from the list of facts a clause may carry, so a streams trend does not ride even with a number. “The line’s own move” stays, because on a market alert the move is the news.

Worked example: the credit moves to the card (2026-09-29)

Owner steer, on an X post that ended “HITS Daily Double has it there.”: “drop them from the text post and make them on the card. The pill is enough.”

The cause was a mandate. attribution.credit_rule said “CREDIT HITS, AND DO IT IN THE SENTENCE”, and gave in-voice examples that named the source (“hits has her tracking ~74k”). A ban on the press-release tag sat beside it. The model obeyed the mandate and moved the name around the sentence. The rule now says only that the card names the data source, so the line names none.

A name in the instruction is a name in the take. The first rewrite said “THE CARD NAMES HITS DAILY DOUBLE AS THE SOURCE”. On the reconciliation lane, Sonnet still wrote “HITS Daily Double projected #3” on 3 of 8 takes. Three more places carried the name to the model: called_shot.describe (the take’s topic line), the block’s per-publisher labels, and the rule itself. Each was reworded by ROLE (“the industry projected #3”), and the rule stopped naming any source. The next run was 0 of 10.

Measured on the real compose inputs of six lanes (compose_market_drop, the desk’s own _CONTEXT + each lane’s framing):

arm	model	takes naming a source
before (main)	terra	23/30
after	terra	0/48
after	Sonnet	1/48

The one Sonnet miss was on the reconciliation lane. The widened has_attribution_tag drops it after one _detag_credit re-ask, which now runs on every desk lane.

Platform boards are the exception, and it is deliberate. On a Spotify or Apple Music board the platform IS the chart (“#1 on spotify’s us chart”), so naming it is the fact, not a credit. Those boards keep their own chart phrasing (platform_chart_phrasing).

Worked example: a mandate to credit, with nobody to credit (2026-10-06)

The newsroom shipped “A third-party tracker puts Drake at 1.7 billion Spotify streams across all credits in September.” The figure was a fan account’s sum of kworb rows.

The cause was a mandate. The SOURCING paragraph of NEWS_CONTEXT told the model to name the most specific source, not “a vague aggregator”, and said a chart-watcher account is a reposter, so “the source to name is the chart or platform behind their post”. This figure has no platform behind it. The model obeyed the mandate with the only words left: “a third-party tracker”. It is the same shape as the HITS credit above.

Before:

SOURCING: when a figure plainly comes from a data authority, name the MOST SPECIFIC upstream source it traces to – the exact platform or chart, not a vague aggregator. [… Songstats is the middleman … a chart-watcher ACCOUNT is a reposter …] Name only a source you’re sure of; if unsure, just state the number. Cite the DATA authority, never whoever reposted it.

After:

SOURCING: a credit is optional. Name a source only when the material names the exact one that measured the number: the platform for a stream count (Spotify, Apple Music, YouTube), Luminate for first-week units, the exact chart for a position, the RIAA or BPI for a certification. Otherwise state the number with no credit. Never write an @handle.

The last sentence keeps the old paragraph’s “never a social-media handle”. It was not the cause, and compose_market_drop has no other handle ban and no gate after it.

Not measured. The session could not pull the runtime keys, so no dry run was made. output_checks.anonymous_credit_hits is the backstop. Measure this reword (before/after, n>=5, on the guild’s GPT model and on Sonnet) before building on it.

Worked example: the label, not the wording (2026-10-17, #4331)

_SHOWS_ITS_WORKING_RULE says “score 0.0”. The Haiku 5.5 scorer did not follow it in two ways:

1. It gave a self-correction the WORKING: label and a score of 0.12-0.55.
2. It gave “Let me correct that: ” the label on only 9 of 48 calls. Its reasons called the lead-in “minor meta-framing” on a clean line.

What shipped is a parse rule, not a prompt edit. _parse_working_score sets the score to 0.0 when the reason opens with the rule’s label. It reads the judge’s own verdict and matches no phrase in the post. In 30 days of production *_scored events, no labeled reason scored 0.6 or more, so no shipped post changes. Shape 1 is fixed. Shape 2 is not.

Seven rewordings were measured, and none shipped. The set was 154 real (post, source) pairs, made by the repo’s dry-run scripts from live data through the lanes’ own source builders, plus 60 working goldens built on 12 of those posts. Each arm ran 2 or 4 times. The noise floor was old against old: 3 posts went ship to drop and 3 went drop to ship.

arm	change	goldens labeled	“Let me correct that” labeled	must-ship “Wait for the weekly”
old	none	121 of 240	9 of 48	0 of 20 labeled
n1	“corrects or redrafts a line, also one the post does not show”	58 of 120	2 of 24	not measured
n5	n1 + “The channel never sees a draft, a request or a search …”	161 of 240	23 of 48	17 of 20 labeled (one window), 2 of 20 (another)
n6	the rule placed FIRST in the prompt	29 of 120	0 of 24	not measured
n8	n5 without “a request”	166 of 240	29 of 48	7 of 20 raw scores under 0.6
n12	n8 + “Only when the post fails this rule, start the reason with ‘WORKING:’”	not run in full	5 of 24	1 of 20

The sentence about how the channel reads a post is what lifts shape 2. The same sentence pulls a finished post that gives the reader advice (“Wait for the weekly”) under the floor. With the parse rule, a reason such as “WORKING: … so shows-working does not fail” also turns into a drop. That is the label-then-pass pattern #4289 saw on HEDGE:. A must-ship post must not newly drop, so no rewording shipped.

Three lessons:

• The rule says “check this FIRST”, but moving it to the top of the prompt cut the label rate from 63 to 29 of 120. The position before the output line is better.
• The run-to-run window matters as much as the wording. n5 labeled the must-ship post 2 of 20 times in one window and 17 of 20 in the next. Run the old arm in the same window as each new arm.
• A parse rule that follows a label is safe only while the judge writes that label on a fail alone. Re-measure a must-ship post whenever the rule’s text changes.

Worked example: details for the chatter lane, and the order that keeps the source (2026-10-17, #4366)

The owner said the KTT2 chatter lane “is not giving me details, it’s too shallow”. In 30 days it composed 187 times and shipped 14. Every compose opened “They’re arguing over …”, gave the subject in one clause and closed on a hedge.

The cause was in the text, in two places. The composer said “tell the room WHAT they’re arguing about”, and the framing said “note it’s a real debate, don’t rule on it”. The model did both. The source was thin too: the lane read only the section page, which cuts the opening post at 280 characters, so a news thread’s names, dates and numbers never reached the model. The fix reads the thread’s first page (the full opening post plus the 6 most-liked replies) and rewords the task to “catch the room up on what happened, with the details, then give your read”. The hedge clause is cut.

The set. 51 real inputs from 28 of the 29 thread episodes in the 30 days, rebuilt from the live thread pages as they stood at each compose time (posts made after the compose were dropped). Production compose model (gpt-5.6-terra, the guild’s market-drop knob) and the production judge (discourse_score, Haiku 5.5). 2 runs per arm per window. Grounding was audited by a separate measurement-only Opus judge.

arm	wording	ship	mean	flips	words	claim stated as fact	invented fact
old (window 1)	as shipped	4/102	0.41	2/51	30	not audited	not audited
old (window 2)	as shipped	13/102	0.41	9/51	29	1/102	6/102
n1	details first, then the read	19/102	0.41	13/51	51	13/102	12/102
n3	take first (“build the take on the details”)	23/102	0.42	9/51	41	34/102	14/102
n3, no replies	n3 without the room’s replies	23/102	0.41	11/51	37	not audited	not audited
n4	n3 + “or the board when they name none (word is)”	16/102	0.40	10/51	42	32/102	23/102
n6 (shipped)	n1 + “the thread claims” + “Add no fact the block does not give”	18/102	0.39	8/51	52	9/102	14/102

Among the posts that would SHIP: old 0/13 claims as fact and 4/13 invented; n6 1/18 and 5/18. Most of n6’s invented flags are small paraphrase slips about the replies; most of the old arm’s are outside knowledge (“Purple Rain”).

Three lessons:

• The order of the task decides whether a claim keeps its source. A take-first wording ships as well but drops the attribution: the model leads with its opinion and the claim becomes the premise. “Details first, then your read” makes the model state the claim with its source before it reacts. Measure grounding next to the ship rate; the self-gate does not see this.
• The old prompt’s grounding came from its shape. “They’re arguing over X” attributes every claim by construction, which is why the old arm almost never states a claim as fact. Removing the shape needs an explicit place for the source (“the thread claims”).
• The replies are a wash on the score (23 against 23). They stay, because they come from the same GET and give the take what the room thinks, which the owner asked for. A reworded attribution clause (n4) made both grounding axes worse; it did not ship.

Worked example: the reason before the score (2026-10-17, #3410 wave 15)

The music desk’s numbers rubric asked for {"score": ..., "reason": ...}. In 14 days, 4 of 119 hard-fail reasons ended “passes, grounded, score high” on a score of 0-0.12. The Ashe forecast line (“forecast to open with ~32K”) dropped 4 times that way. The model wrote the score first, then reasoned to the opposite answer, and the parser kept the score.

What shipped. The rubric’s output line now puts the reason first. The token budget is 400 (_DESK_SCORE_MAX_TOKENS), because a reason-first answer that truncates loses its score. The UNGROUNDED line also says “Start the reason with ‘UNGROUNDED:’”.

The set. 332 numbers-rubric posts: 51 from the dry-run scripts (live data, the lanes’ own source builders), 140 real production tracking posts with the source rebuilt in the luminate.format_reading shape, 120 must-fail posts made from real posts (a wrong figure, or “though that figure is unconfirmed”), the 4 real disagreement cases (2 sources rebuilt exactly, 2 from what the reason quotes), and 17 eval fixtures. Haiku 5.5, 4 runs per arm.

arm	change	correct verdicts	posts worse	flips between runs	UNGROUNDED: label on the 60 wrong-figure posts
old	none	1303 of 1328	-	1-6	237 of 240
A	reason first	1325 of 1328	0	1-3	140 of 240
A2 (shipped)	A + “Start the reason with ‘UNGROUNDED:’”	1328 of 1328	0	0	209 of 240
B	old + re-score when a Haiku classify() reads the reason as PASS	same as old	1	4	-

Three lessons:

• The disagreement was mostly gone before this change. The old prompt shipped all 4 real cases on 4 of 4 runs, and 0 of 21 low production scores after the #4321 deploy showed the shape. The gain that remains is consistency: the old order dropped clean forecasts on 1-4 of 4 runs, and it shipped a fabricated figure (“~11K” against “~5K”) on 3 of 4 runs.
• The label came from the order. The rubric never asked for UNGROUNDED:. The judge wrote it because the score came first. With the reason first it wrote prose, and the gating golden “an invented forecast fails on an UNGROUNDED: reason” went red 3 of 3. The HEDGE line already says “Start the reason with ‘HEDGE:’”, and that label held at 240 of 240 in every arm. The same sentence on UNGROUNDED brought the label back.
• Measure each rubric on its own. On the release rubric the reason-first order shipped “eternal sunshine is confirmed for July 31” against a block that dates the album petal (3 of 4 runs, against 0 of 4). So the release rubric keeps the score first.

The re-score arm did not ship. Its classify() check said PASS on a WORKING: fail and on a HEDGE: fail, so it would re-roll real fails, and the real disagreements it targets no longer occur.

Worked example: a wrong qualifier, a separate judge (2026-10-17)

A wire posted “Tate McRae hits No. 1 on the Global Spotify chart for the first time ever”. The No. 1 was true. “First time ever” was false, because “greedy” held Global No. 1 before. The web research said so. The X research repeated the wire’s words. The Haiku verify judge (verify_music_claim) answered verified.

All arms below read the same live research, saved once, on the production models.

arm	change	Tate “first time ever” refuted or cut	side effects
old	none	0 of 5	none
A	“an earlier No. 1 where the claim says first No. 1” added to the contradicted examples	0 of 5	not measured
B	A + “the record includes every qualifier the claim states” in verified	1 of 5	not measured
C	a separate earlier_instance field on the verify prompt	5 of 5	set on 1 of 5 plain “debuts at No. 1” claims; Bruno Mars verified 2 of 16 (old 9 of 16)
D	C, asked only when the claim states a first (not “first of”), max_tokens 180 to 260	4 of 4	Usher verified 9 of 16 in both arms; Bruno Mars (“the first of five concerts”) back to the old prompt
E	D generalized to every qualifier, with a trimmed_claim field	2 of 5 cut, 3 of 5 dropped	cuts unreliable: one cut the FACT and kept the qualifier, two cut “ever” and kept “for the first time”
F (shipped)	the verify prompt unchanged; a separate Sonnet call, check_claim_qualifier, only for a claim with a qualifier	4 of 4 cut to “#1 on the Global Spotify chart”	7 true or unaddressed qualifiers kept 4 of 4 each
G (shipped later)	F + “a qualifier that reads two ways is judged on the broader reading”	the pop desk’s copy of the wire tweet (“debuts at #1 … for the first time ever”), 4 fresh research samples x 3: cut 12 of 12 (F: 9 of 12)	“first time in her career” wording: 6 of 12 both arms (the research decides); Usher’s “first tour to gross over $200 million” now cut 4 of 4 (false as a first-ever); Ariana’s cut became a drop 1 of 4

Four lessons:

• Rewording a definition did not move Haiku (arms A and B). The judge’s own note quoted the conflict and it still answered verified. A narrow yes/no question did move it (arm C).
• Each field added to a shared judge prompt moved verdicts on claims it did not concern (arms C and E). A separate call keeps the old prompt byte for byte for every claim.
• An ambiguous qualifier is the case the research decides. When the research reports only the narrow true reading (“her first No. 1 debut”), the judge kept the claim and the compose then wrote the broad false one (“her first Global Spotify No. 1”). Arm G names the broad reading.
• A model’s cut of a claim needs a check in code. qualifier_trim.is_cut accepts a cut only when every kept word is in the claim in order and every removed phrase carries a qualifier. That one rule stops arm E’s worst reply: a cut of the fact itself.

Worked example: a ranking with no number, a separate judge (2026-10-24)

The newsroom posted “JENNIE has the second longest-running active streak among female artists on the Global Spotify chart” with a “2nd” card. The verify note said “JENNIE at 1,113 days”. The owner’s rule: a ranking states its stat. A new Haiku call, name_ranked_stat, reads the claim and the research. It is a separate call for the same reason as the qualifier check: a field added to the verify prompt would move verdicts on claims it does not concern.

The set: 400 real newsroom lines from the 7 days to 2026-10-24 (Axiom music_news_scored post_preview), 132 of them with a ranking word, read with no research.

arm	prompt	ranks=true	wrong flags
first draft	“ranks = true when the claim places the subject by a superlative, an ordinal place or a top N … and does not state the subject’s own value”	76 of 132	most: count ordinals (“his 44th song past 100 million”) and lines that already state the number (“passed 1.8 billion streams, the most-streamed album”)
shipped	“ranks = true only when BOTH hold” + a list of the false cases (a stated number, a count ordinal, a chart position, no quantity ranked)	18 of 132	2 borderline (“only the second 1970s song to reach the mark” with “400 million” in the claim)

The live case returned “1,113 days” in both arms. A code check backs the prompt: mn.stat_grounded refuses a number the text does not carry, and the number must have the stat’s unit word right after it, so a chart name (“Hot 100”) never grounds “101 weeks”. A word-list skip for claims that already state an amount was tried and removed (Codex review on #4755): it also skipped “behind Taylor Swift at 1,812 days”, which states another act’s amount and not the subject’s. The 18-of-132 count above was measured without that skip. The extra search was measured on 9 flagged lines: with no date limit it found 7 but read a song’s “200 days” as JENNIE’s streak; limited to the last week it found 3, all plausible. The week limit shipped. The record compose, given the claim with the stat, wrote “1,113 days” 4 of 4 times (2 Sonnet, 2 gpt-5.6-terra).

Worked example: the reason before the verdict, and the words that open it (2026-10-17, #4443)

The topic-dedup judge (topic_duplicate) answered {"duplicate": ..., "reason": ...}. In 10 days, 27 of 736 shipped overturns had a reason that concluded “restates post N, no new development” under a false verdict. Most of those reasons began “X is a new subject” and found the matching post later. The verdict followed the first clause.

The set. 307 real production verdicts, 2026-10-07..10-17: the 27 bad overturns, 200 random other overturns, and 80 topic_dedup suppressions. Each candidate was judged against its real history (the post_dedup_history rows from 10-14 on, the *_posted previews before that) through the real evidence order of arbitrated_match. Haiku 5.5, 2 runs per arm. Opus 5.5 labelled the 132 cases where the arms disagreed, with the judge’s own policy text.

arm	change	correct (of 614)	repeats shipped	stories dropped
old	none	476	64	74
new	reason first, same example reasons	461	14	139
D	reason first, the example reasons open with the compared post	495	3	116
E	D, the false example first	518	6	90
F (shipped)	E, the false example is a chart position, not a bid	533	9	72

Three lessons:

• Reason-first moves the lever to the example reasons. With the verdict first, the verdict token was the first thing the model wrote. With the reason first, the first words of the example reason are. The shipped example began “restates the LeBron signing”, and arm “new” opened most of its reasons with “restates” and then answered true. It fixed the bad overturns and dropped almost twice as many real stories.
• An example far from the traffic does not teach its rule. The prompt says a new number is a development, and the false example was a transfer bid. On chart positions and forecasts, arm E still wrote “the same story with a different number”. The chart-position example (arm F) brought the dropped stories back to the old level.
• The self-labelled set lied. The first measurement counted every overturn whose old reason said “new” as a correct ship. Opus found 17 of those were repeats, so the scoring needed an independent label on every disagreement.

F changes its answer between runs on 29 of 307 cases (old: 16). The cost is about 11 more output tokens per call (mean 95 against 84), no added call.

Worked example: the words a cut leaves dangling (2026-10-18, #4465)

The qualifier judge (check_claim_qualifier) cut only the qualifier words. On “Tate’s ‘Housewife’ is her first song ever to hit No. 1 …” it wrote “is her song to hit No. 1” 8 of 8 times. The claim is true, but the sentence is broken. The music drop ships the cut with no compose after it.

The cause was existing text. The trim line said “Cut the whole qualifier phrase … Keep every other word exactly as written”. The judge obeyed it.

The set. 18 real claim shapes, all with a false qualifier unless marked true:

• 5 from the #4459 and #4423 dry runs and the incident wire;
• 4 true qualifiers from #4459;
• 5 held out that no prompt example names (Bad Bunny, Gaga, Kendrick “luther”; Olivia and SZA “Kill Bill”, both true);
• 4 from the Codex review of #4468 (Kendrick “HUMBLE.”, Adele “6 Grammys”, Tate “first female artist”, Tate “first track to reach”).

qualifier_research ran once per claim, and every arm read that saved text. Sonnet, n = 8 per case per arm. “full” = the cut is a full grammatical sentence with the claim’s meaning.

arm	change to the trim line	Tate drop	Ariana wire	other rows	true kept
old	none	cut 8, all broken	cut 8 (“It becomes her #1 hit.”)	Bad Bunny 4 broken + 4 drop; HUMBLE. 8 broken; Kendrick wire 7 cut + 1 drop	48 of 48
A	old + “must read as a full sentence; cut broken words too”	3 full, 3 broken, 2 drop	cut 5, drop 3	not run	32 of 32
B	reworded around “a full sentence”, the Tate example	8 full	cut 2, drop 6	not run	32 of 32
C	B + “Keep every sentence of the claim”	8 full	cut 2, drop 6	not run	32 of 32
D	old + “and the words left dangling without it”, “‘is her first song ever to hit No. 1’ becomes ‘is No. 1’”	8 full	cut 6, drop 2	Kendrick wire 7 cut + 1 drop	48 of 48
E	D + “Never cut a whole sentence”	13 full, 3 broken (n = 16)	cut 13, drop 3 (n = 16)	not run	not run
F	D + “‘It becomes her first-ever #1 hit.’ becomes ‘It becomes her #1 hit.’”	8 “is No. 1”	cut 8	Bad Bunny 8 “‘DtMF’ is No. 1” (false now); HUMBLE. 1 broken + 7 drop	48 of 48
G1 (shipped)	F, the first example becomes “‘is her first song ever to hit No. 1’ becomes ‘hit No. 1’”	4 full, 2 broken, 2 drop	cut 8	Bad Bunny 6 full + 1 broken + 1 drop; HUMBLE. 3 full + 2 broken + 3 drop; Kendrick wire 8 cut	48 of 48
G2	G1 + “’’ when no cut leaves a full sentence with the claim’s meaning and tense”	4 full, 2 broken, 2 drop	not run	HUMBLE. 1 full + 6 broken + 1 drop	not run
G3	G1, the example with its subject (“Her song hit No. 1.”)	2 full, 6 broken	cut 8	Bad Bunny 5 full + 3 broken; Tate female 3 full + 5 drop	not run

Across all 12 false claims, G1 against old: 68 cuts against 67, 35 full cuts against 20 (not counting the Ariana-shape “her #1 hit”), 9 broken cuts against 24. No arm kept a false qualifier. The Adele and “first track to reach” claims have no full-sentence cut in their own words; every arm drops or breaks them.

Four lessons:

• “Full sentence” invites a whole-sentence cut. Every arm that named a full sentence (A, B, C) also made the judge cut “It becomes her first-ever #1 hit.” whole. That cut is good prose, but is_cut rejects it (four content words), so the story dropped. A rule against it did not fix it: “Keep every sentence” gave no change, and “Never cut a whole sentence” still dropped 3 of 16. A second example fixed it.
• An example teaches its exact output, including its tense. F’s “becomes ‘is No. 1’” fixed the Tate cut 16 of 16, and it also made “‘DtMF’ is No. 1 on the Global Spotify chart” 8 of 8, for a song that topped the chart in 2025. The cut is grammatical and passes is_cut, but it states a current No. 1 that is false. G1’s “hit No. 1” keeps the claim’s own verb, so a past No. 1 stays past. Codex found this shape in review; the run confirmed it.
• Name the shape, not the goal. “The words left dangling” and one example moved the Tate cut more than any “full sentence” rule did.
• An example that is a test case needs held-out cases. The second example is the Ariana claim. The held-out Gaga and Kendrick claims have the same shape and other words, and they kept their cut 16 of 16.

“It becomes her #1 hit.” is grammatical but weak. The better cut removes the sentence. That needs an is_cut rule that accepts a whole removed sentence with a qualifier in it. If that rule lands, re-measure arm B against G1: G1’s second example then teaches the weaker cut.

Worked example: a sourced record, an invented record, and the order of the reason (2026-10-18, #4457)

The cinema desk’s news rubric (cinema_desk_score, kinds release / awards / tv_news / tracking) asked for {"score": ..., "reason": ...}. On real lane pulls it had two faults:

1. It dropped a record the source states. “Apple TV posted its largest weekly viewership ever” scored 0.12-0.45 against a source line that says it.
2. It shipped a record the source does not state. “renewed Slow Horses after its most-watched season ever” scored 0.78, and the reason said “a fabricated superlative”. The score did not follow the reason.

The parser and the token budget were not the cause. All replies had a closed JSON object, the longest about 1,050 characters against a 250-token budget. The issue’s WORKING: reasons were the judge using the label for its own doubt (“the post’s own text is fine, but … cannot be verified”). In most of them the judge wrote 0.0 itself, so the _parse_working_score clamp decided almost none of them.

The set. This week’s real pulls, through the lane’s own queries and story builders, frozen to one file: two tracking pulls of the same day, one awards pull, one tv_news pull, and a live TMDB release blob with 3 takes the guild model composed on it. 2 true cases add one sourced record line to a real pull. 4 goldens add a record the pull does not state. scripts/dryrun_cinema_desk_score.py. Haiku 5.5, n = 8 per case per arm, old arm re-run in every window. The window swing is large: across 8 windows the old arm shipped the Apple case 0 to 5 times in 8, and leaked invented records 2 to 15 times per window.

arm	change	true (ship)	invented (leak)	control (ship)	label-then-pass
old	none	3-13 of 16	2-15 of 32	29-39 of 40	0
B	ground-truth label names the SOURCES headlines	7/16	12/32	40/40	0
C	hard fail names ‘first’ / ‘biggest’ / ‘ever’, “one it does state is grounded”	8/16	1/32	37/40	0
W	“the WORKING: label is only for the post’s own words”, after the output line	15/16	2/32	37/40	labels doubled
R	reason before the score, budget 400	14-16 of 16	2-4 of 32	40/40	0-2 of 128
R + C3	R + the hard fail names the record words	10-13 of 16	0/32	33/40	6-7 of 128
R + X	R + example reason “every claim in it is in the ground truth”	16/16	2/32	40/40	0
R + X + C2 (shipped)	R + X + “record claim” in the existing hard-fail list	16/16; n=16: 32/32	0/32; n=16: 2/64	39/40; n=16: 80/80	0-1

At n = 16 in one window the old arm shipped true 20/32, leaked 27/64, and shipped control 68/80.

Four lessons:

• The reason-first order fixed the score-follows-reason fault here too (as in #3410 wave 15). With the score first, the judge’s reason said “unsourced superlative” under a 0.78. With the reason first, the score followed the reason.
• Naming the words is a lever with a cost. C and R + C3 list ‘first’, ‘biggest’ and ‘ever’ and stop every leak. They also make the judge check every plain post for a record, and with the reason first it opens on that check: “WORKING: no, the post is clean”, which the clamp turns into a drop. Two words, “record claim”, in the existing list (C2) did most of the work without that cost.
• Naming the label raises its use. W says when not to use WORKING:, and the judge wrote the label twice as often. That is the #4331 lesson from the other side.
• A figure only a cited headline states still drops: 0 to 2 of 8 ship per case in every arm, 0 of 32 for the shipped arm at n = 16. The judge calls the answer “the tracking block” and the SOURCES list something else, and a reworded label (B) or tracking beat did not move it. When the pull’s answer states the figure, the same take ships 5 to 8 of 8 in every arm (16 of 16 for the shipped arm at n = 16), so the tracking lane can ship. That pull also carried a Friday result for the same film. Prefer absent over invented: this drop is left in place, and the pull is the place to fix it.

The numbers rubric (box office / streaming / scores) keeps the score first. It was not measured here.

Worked example: when a ranking ordinal is wrong (2026-10-18, #4472)

Since #4471 a ranking ordinal (“the 2nd electronic artist to reach 35B”) reaches the qualifier judge (check_claim_qualifier). The prompt had no rule for one. The #4471 dry run dropped two real claims on wrong reasoning:

• Calvin Harris, “the 2nd electronic artist to reach 35B”: “David Guetta is ahead, so Harris is not the 2nd”. One earlier act agrees with “2nd”.
• Asake, “the 6th Afrobeats artist to pass 4B”: “the 5th Nigerian artist” was read as a refutation. That is a different group.

The set. 41 claims, each with 4 stored qualifier_research samples (real Perplexity), read in turn by every arm. Sonnet, n = 8 to 64 per case per arm.

• Tuning: Calvin, Asake (true); Avatar “the 3rd film to pass $2.5 billion” and Taylor Swift “the 5th female artist” to 100B (false: both were first).
• Held out: PLK, Chase Atlantic, Destiny’s Child, “Dracula” “the fifth song this decade” (real wire claims, true); Endgame “second film to pass $2.5 billion” and Billie Eilish “second artist to win all four general field Grammys in one night” (true); Titanic “2nd film to gross $1 billion” and Billie Eilish “3rd artist” (false).
• Regression (#4468 shapes): Tate newsroom, wire and drop; Ariana wire; SZA; Bad Bunny (false); Beyoncé, Taylor AOTY, Mariah, Tate “biggest-ever debut”, Cardi B (true).
• 16 more real claims from the coordinator, among them Tate “debuts at No. 1 on global Spotify for the first time in her career” and Usher “first tour to gross over $200 million” (both false).
• Observed only (truth unknown): Tems, “Sweet Child O’ Mine”.

Ordinal claims.

arm	change	Calvin dropped	Asake dropped	Avatar kept	Dracula dropped
old	none	8/16	3/16	12/48	0/48
O1	rule “N-1 earlier X agree; another group does not refute” + “ranking ordinal” in the word list	2/8	0/8	0/8	2/8
O2	O1 rule only	2/8	0/8	1/8	0/8
O3	O2 + “‘the 3rd K-pop act’ is right when two K-pop acts topped it first” + “a count the research does not give is not wrong”	0/16	2/16	0/16	4/16
O4	O3 + the word-list change	0/16	0/16	0/16	3/16
O5	the rule as a count (“wrong only when the research names N or more X … or shows the subject did it earlier than Nth”; “naming fewer earlier X does not refute it”) + the word-list change	0/16	0/16	4/48	0/48
O6 (shipped, ordinal claims only)	O5 without the word-list change	0/16	0/16	0/48	3/48
O9	O6, “a ranking ordinal of 2nd or later”	0/48	0/16	3/48	1/48
O11	O9’s text at the end of the paragraph	9/48	0/16	0/48	11/48
O12	O9 + “or ‘for the first time in her career’” in the broader-reading example	0/48	0/16	0/48	1/48

Every other true ordinal (PLK, Chase, DC, Endgame, Billie “second”) was kept 80/80 and every other false one (Taylor “5th”, Titanic, Billie “3rd”) cut or dropped 48/48 in every arm from O5 on.

The regression the ordinal rule caused. The coordinator’s claims showed it. Every wording of the rule made the judge read some “first” claims narrowly, as the subject’s own history:

arm	Tate “first time in her career” kept	Usher “first tour over $200M” kept
old	0/64	6/64
O6	8/32	0/32
O7 / O8 (O6 moved later in the paragraph)	5/32 / 6/32	not run
O9	3/64	2/8
O10 (O9, “(1st, for a claimed ‘3rd’)”)	8/32	not run
O11	0/40	2/8
O12	0/40	16/64
O13 (O9 + “A ‘first’ is judged by the rules above, not by this count.”)	5/64	0/32
O14 (O13 + O12’s example)	0/32	7/32

The kept reasons read “research confirms Housewife is her first No. 1 debut” and “research supports it as Usher’s first $200M+ tour”. No wording fixed both.

Shipped: the rule only for a claim with a ranking ordinal. check_claim_qualifier adds O6’s rule only when qualifier_trim._rank_matches(claim) finds one. Every other claim keeps the old prompt byte for byte (checked in code for all 41 claims: 14 get O6, 27 get the old prompt). So the ordinal claims behave as O6 above, and the #4468 and coordinator claims behave as old. A real run of the shipped method on stored research (n = 4) gave Calvin kept 4, Avatar dropped 4, Tate career cut 4, Usher cut 4.

Four lessons:

• A rule about what refutes must name what does NOT refute. O1 and O2 said “N-1 earlier X agree” and still dropped Calvin: the judge read “Guetta is ahead” as a refutation. Stating the count both ways fixed it.
• More attention to ordinals means more drops of unconfirmed counts. O3 and O4 dropped the true “Dracula” claim because the research named one earlier song and “did not substantiate” the count. The count rule stopped it.
• A rule for one claim shape can move verdicts on another. The ordinal rule never names “first”, yet it made the judge read “first time in her career” and “first tour” narrowly. Nine wordings and placements traded one of those claims for the other. When the shape is detectable in code, gate the rule on it, and the other claims keep the old prompt exactly. This is the same lesson as the separate qualifier call (arm F of the 2026-10-17 example).
• Test with claims the author did not pick. My first set had no “first … in her career” or “first tour” claim, and O6 looked clean on it. The coordinator’s 16 claims found the regression.

Worked example: a time span, “joins X as the only”, and an unconfirmed “most” (2026-10-18, #4512)

In its first ten hours the qualifier judge (check_claim_qualifier) dropped three true claims in production, one of them five times:

• “Biggest opening weekend for a non-music documentary in over 2 decades”: “Fahrenheit 9/11 opened to $23.9M in 2004”. 2004 is 22 years back, outside the span.
• Taylor Swift “joins Madonna & Rihanna as the only artists … to achieve 30 top ten singles”: “Rihanna had already reached 30”. The claim names Rihanna.
• Jungkook “his 7th song to reach this milestone, the MOST among K-pop soloists”: “V had five … the research does not establish that Jungkook’s seven is the record”. Five is fewer.

The set. 32 claims, qualifier_research stored 4 times each (real Perplexity and kworb), every arm read the same texts, Sonnet, n = 16 per claim per arm unless marked:

• the three production claims;
• held out, true: Barbie “the biggest opening weekend of 2023”, Billie Eilish “joins Christopher Cross as the only artists to win all four general field Grammys”, Drake “the most Hot 100 entries”;
• held out, false: Inside Out 2 “the biggest animated opening in over 10 years” (Incredibles 2, 2018), Joker: Folie à Deux “the biggest October opening in 5 years”, Kendrick Lamar “joins Christopher Cross and Billie Eilish as the only artists to win Record and Song of the Year in the same night”, Ed Sheeran “the most streamed song in Spotify history”;
• the goldens: the four Tate / Bad Bunny claims, Avatar “3rd”, the #4468 set (the Gaga, “luther”, Tate “first female artist” and Tate “first track” texts are rebuilt from their shapes);
• the controls: Slayyyter, KATSEYE, PopBase, Chappell, sombr, Olivia Dean, Calvin, Asake.

Time span (the doc claim kept, of 16 per run):

arm	change	doc kept	Inside Out / Joker kept
old	none	4, 4	0 / 0
W1	“covers only that span … an instance from before the span agrees”	11	0 / 0
W2	W1 + today’s date + “a bigger one from more than ten years ago agrees”	10 (2 unread)	1 / 0
W3 / W4	W2 reworded, “count the years” / “13 years back”	8 / 4	0 / 0
W5	“‘the biggest in over N years’ says nothing bigger is less than N years old”	15, 15, 14	0 / 0
W6	“the span is counted back from today”	12	0 / 0
W7	W5 + “nor a bigger one outside the group the claim names”	8, 7	not run
W8	W5 + “so the claim is right”	12, 12	not run
W9	code gives the span years (“2006 to 2026”)	4, 4	0 / 8
W5R	W5, the reason before the verdict	13	0 / 5
W9R (shipped)	W9, the reason before the verdict	16, 15, 15	0 / 8

“Joins X and Y as the only” (Taylor kept or cut, of 16): old 7 and 3; J1 (shipped) “X and Y agree with the claim; ‘only’ is wrong only when the research names another holder the claim does not name” 16; J1R (reason first) 9; J2 (J1 + “acts with fewer do not refute it, and an ‘only’ the research does not confirm is not wrong”) 13 and 12.

“Most”: old kept Jungkook 12 of 16. M1 (shipped) “a ‘most’ is wrong only when the research names an act with MORE; acts with fewer agree with it, and a ‘most’ the research does not confirm is not wrong” kept 16 of 16, and dropped the false Sheeran claim 16 of 16. NW1, a general rule (“WRONG needs a specific counterexample, and ‘why’ names it”) for every claim, did not fix the doc or Taylor claims, and it kept the false Tate “first female artist” 8 of 16 against 4 for old. So each rule is added only when code finds its shape: qualifier_trim.time_span, joins_as_only, states_most. Every other claim keeps the old prompt byte for byte (checked in code for all 32 claims).

Shipped, through the real check_claim_qualifier (main and shipped read the same stored research; the three claims also ran twice more as arm S, which is byte-identical to the shipped prompt):

claim	truth	main	shipped
doc “in over 2 decades”	true	kept 8/32	kept 47/48
Taylor “joins … as the only”	true	kept or cut 10/32	kept 60/64 (J1 arm + shipped + S)
Jungkook “the MOST among K-pop soloists”	true	kept 28/32	kept 48/48
Barbie, Billie, Drake (held out)	true	kept 48/48	kept 48/48
Inside Out 2, Sheeran (held out)	false	dropped 32/32	dropped 32/32
Joker “in 5 years” (held out)	false	dropped 16/16	kept 8/16
Kendrick “joins … as the only” (held out)	false	cut or dropped 16/16	kept 4/16
Tate / Bad Bunny goldens, Avatar, the #4468 set, the controls			the old prompt; same within noise

Three lessons:

• The judge could not place a year in a span, whatever the wording. Eight wordings moved the doc claim between 4 and 15 of 16, and the reasons kept calling 2004 “inside the ‘over 2 decades’ span”. When code gave the years, the judge wrote “inside the 2006-2026 span” about 2004 too, until the reason came before the verdict. With the verdict first, the judge answered wrong and then argued for it. This is the #4443 lesson again: reason-first makes the verdict follow the reason.
• Reason-first is not a general fix. On the joins claim the reason-first order was worse (9 of 16 against 16 for verdict-first). So the order changes only with the span rule.
• A held-out false claim can measure the rule’s limits instead of the rule. Joker’s claim is from 2024, and the shipped span counts from today, so 2021-2026; two of its four research samples name only 2018 and 2019 openings. The Kendrick claim is kept on one research sample that itself says “Cross and Eilish … the only two”. Both are kept on research that does not name an instance inside the claim. A production claim is a new post, so counting from today fits it.

Two review wordings, measured and not shipped (Codex review on #4516):

• The span rule as “an instance that breaks it (a bigger one for a ‘biggest’ or ‘most’, an earlier one for a ‘first’)”: doc kept 43/48, against 47/48 for “a bigger or earlier instance”. The case it guards was already held: Barbie “the biggest opening of 2023” has an earlier, smaller 2023 opening in its research (Super Mario), and the shipped wording kept it 16 of 16 in every run.
• The joins rule + “or shows that the subject, X or Y did not do it”: Taylor kept 12/16. The shipped joins rule already cuts or drops the new golden (Billie Eilish “joins Christopher Cross and Adele as the only”, Adele never won all four in one night) 32 of 32, and 32 of 32 more through the real method: the general “shows it is false” rule covers a named holder who did not do it.

Worked example: the shape gates move from regex to one model read (2026-10-18, #4522)

The #4512 rules are added only when a claim has the shape: a time span, “joins X and Y as the only”, or a maximum “most”. Three regexes decided that (qualifier_trim.time_span, joins_as_only, states_most). The owner rule of 2026-10-18 keeps a regex only for an exact format, so this site was the pilot of #4522.

The defect. On 62 real newsroom claims of 30 days that touch a shape, the regex read 30 right on all three gates. It missed “joins The Beatles as only acts” (no “the”), “Sets new record for most No. 1 albums” and “most #1 hits in US Pop radio history”. It fired the “most” rule on “the 4th most-streamed female album” and on “rises to No. 58 among the most-viewed music videos”. The “most” rule then tells the judge that only an act with MORE refutes the claim, which is false for a 4th place.

What changed. Code keeps the exact part: span_phrases lists the claim’s time phrases, and phrase_span computes their years. One cached Haiku read (qualifier_shape) gets the claim and the numbered phrases. It answers which phrase limits a qualifier, and the joins and most questions, as one JSON line. The old patterns are in its task as examples. On an error or an unreadable reply, the regex gates decide.

The set. 172 hand-labelled claims: the 33 harness claims (the three #4512 claims among them), 63 detector probes (the tests/test_qualifier_trim.py cases, the coordinator’s #4516 probes and two known misses), the 14 distinct qualifier_check matchups since 2026-10-17 (all that exist, cut at 80 characters by the event), and the 62 newsroom claims. Three runs per arm.

arm	span	joins	most	all three	real claims (109)	flips	p50 / p99 ms	$ per 1k
regex	170/172	170/172	142/172	138/172	77/109	0	0	0
Jev (TypeSafe.ask, Choice + 2 Noul)	516/516	509/516	438/516	431/516	300/327	7	789 / 913	0.031
Haiku, wording 1	514/516	516/516	505/516	503/516	319/327	2	584 / 1812	0.154
Haiku, wording 2 (shipped)	510/516	513/516	513/516	504/516	324/327	0	589 / 1174	0.154

Wording 1 gave “spent most weeks at No. 1” as a quantity. Haiku then read “extends record for most weeks” and “ties ‘Luther’ for most weeks” as quantities too (3 real misses). Wording 2 adds “ties the record for most weeks at No. 1” as a maximum and gives the quantity example as “spent most weeks of the year at No. 1”, under “means ‘the greater part of’”. Its one real miss is “top two most streamed songs … this year”: no span rule, so the old prompt (absent, not invented). Its probe misses are short fragments with no subject (“her first album in 5 years”, “alongside Drake as the only rappers”).

Jev read every “biggest” as a maximum “most” (13 claims, all 3 runs). That is the wrong direction: the “most” rule would reach the doc, Barbie, Inside Out and Joker claims.

Three lessons:

• A false example pulls its neighbours. One quantity example (“spent most weeks”) moved three real records the same way. Name the meaning (“the greater part of”) and give the nearest true case beside it.
• Keep the exact part in code even with a model. Haiku picks a phrase by number, and code computes the years. The model never writes a year, so a misread can only drop a rule, not state a wrong span.
• Check that the converted gate leaves the judge alone where it agreed. The regex reads all 33 harness claims the same as Haiku, so their judge prompts must not change. The run below checks it on every call.

The #4512 table, end to end. Fresh qualifier_research (4 samples per claim, real Perplexity and kworb), the real check_claim_qualifier with the live shape read, n = 16 per claim. Each of the 528 judge prompts was compared with the prompt the regex gates give on the same research: all 528 were byte-identical. So each rate below is main’s rate on this research.

claim	truth	kept	cut	drop	no verdict
doc “in over 2 decades”	true	14	0	1	1
Taylor “joins … as the only”	true	15	0	1	0
Jungkook “the MOST among K-pop soloists”	true	8	0	6	2
Barbie, Billie joins, Drake most (held out)	true	48	0	0	0
controls: Slayyyter, KATSEYE, PopBase, Chappell, sombr	true	80	0	0	0
Olivia Dean / Calvin / Asake	true	8 / 12 / 0	8 / 0 / 16	0 / 4 / 0	0
Tate ever / wire / career, Bad Bunny	false	0 each	16, 16, 16, 13	0, 0, 0, 3	0
Inside Out 2, Sheeran, Avatar	false	0	0	16 each	0
Joker “in 5 years”	false	4	0	12	0
Kendrick joins / Billie-Adele joins	false	4 / 0	5 / 8	7 / 8	0
#4468 set (9 claims)	false	Tate “first female” 4, others 0

Jungkook is under 15 of 16 on this research, with main’s own prompt. Re-judged per research sample, 4 times each: samples 0 and 2 kept 8 of 8. Sample 1 dropped 4 of 4: that research itself calls the “most” false. Sample 3 gave 2 drops and 2 no-verdicts, and one drop’s reason says “the ‘most’ is unconfirmed rather than contradicted, so it is not wrong”. The shape read was right on all three claims, and the “most” rule was in the prompt. The rest is the judge and the research, not the gate.

Worked example: the “most” rule puts the reason first (2026-10-18, #4537)

The defect. On fresh research, main’s judge kept the true Jungkook claim (“his 7th song to reach 400M, the MOST among K-pop soloists”) 12 of 16 times. All misses came from one research sample. On that sample alone, main kept it 7 of 32 times (19 drops, 6 no verdicts). The reasons contradicted the verdicts: “V had five solo tracks above 400M, but Jungkook’s 7 would exceed that, so no act with MORE is named”, with wrong: true.

The cause, from the raw replies. The judge wrote {"wrong": true, ...} first. Its reason then found no act with more than seven. Then it wrote “Correction, since the above is internally inconsistent” and a second line with wrong: false. The parser takes the text from the first { to the last }. When the 200-token limit cut the second line before its }, that text was the first object alone, so the parser read wrong: true and the claim dropped. When both objects closed, the text held both, did not parse, and the reply gave no verdict. The verdict came before the reasoning, as in the #4516 span rule. The fix is in the prompt, not the parser: with the reason first, the judge wrote one verdict, and that verdict followed its reason.

Arms, on the same stored research (fresh1: 4 samples, 8 judges each; the hard sample: 32 judges):

arm	change	fresh1 kept	hard sample kept	Drake “most” (true)	Sheeran “most” (false)
main	none	27/32	7/32	32/32	dropped 32/32
R (shipped)	the reason before the verdict, only with the “most” rule	32/32	32/32	32/32	cut or dropped 32/32
C	a counterexample field; a WRONG with an empty one reads as no verdict	32/32	32/32	32/32	dropped 32/32
RC	R + C	32/32	32/32	32/32	dropped 32/32

C fixed the claim because the judge answered wrong: false. No reply took the empty-counterexample path, so that code path had no measured use. R is the smaller change, with no new field and no parser logic, so R shipped.

End to end, the real check_claim_qualifier with the live shape read, n = 16 per claim on fresh1 (main’s rates in brackets): Jungkook kept 16 (12); doc span 16 (16); Taylor joins 16 (16); Drake 16 (16); Sheeran dropped 16 (16); Tate ever / wire / career and Bad Bunny cut or dropped 16 each (16 each); Avatar dropped 16 (16); the #4468 set cut or dropped 16 each (16 each); the controls within noise (Olivia Dean kept 15 (16), Asake kept 9 (8), with the same prompt). Joker “in 5 years” kept 16 (16) and Kendrick joins kept 4 (4): these are the #4516 limits, and their prompts did not change. A new fresh set (fresh3, 4 samples): Jungkook kept 16/16, Drake 16/16, Sheeran dropped 16/16. Only Jungkook, Drake and Sheeran get the “most” rule from the live shape read, so the other 30 claims send main’s prompt byte for byte.

Two lessons:

• Read the raw reply before you change a rule. The reasons alone looked like a judge that misread the rule. The raw text showed a judge that got the rule right one line too late.
• Reason-first again fixed a contradiction between the reason and the verdict. It still is not a general fix: on the joins rule it was worse (#4516), so each rule gets the order that its own measurement supports.

Worked example: a word list becomes the candidates, and a NO example pulls its neighbours (2026-10-18, #4522)

qualifier_trim.has_qualifier decides whether a claim pays for the qualifier research and the Sonnet judge. It was a word list (first, ever, only, most, best, record, …) plus the ranking regex. This is the second #4522 site.

The defect. On 1,000 distinct newsroom claims of 30 days, the list fired on 168. Hand labels put 13 of them at no qualifier: “58-63k first week”, “11K first-week units”, “played the first night of a two-night concert”, “Best Rap Album”, “Best Original Song”, “from lead songs only”, “our most recent chart”, “Greatest of All Time Hot Latin Songs”, “Record Store Chart”. Its misses were rare: 3 of 832 (“Every studio album since 2000 has debuted at #1”, “top new debut”).

What changed. The list stays the fast path. qualifier_trim.qualifier_candidates lists the words outside quoted titles; a claim with none is False with no call. A claim with candidates gets one cached Haiku YES / NO (claim_has_qualifier, purpose qualifier_present), with the claim and its candidates as input. No verdict is True, the list’s answer.

The fail direction decides the metric. A NO skips the judge, so a NO on a real qualifier ships an unchecked claim. A YES on a false hit only costs a search and a judge call. So the count that matters most is true qualifiers read as NO, not overall accuracy.

arm	labelled hits right	false-hit reads dropped	true qualifiers read NO	goldens	flips	p50 ms
word list	130/143	0/13	0	41/41	0	0
Jev (TypeSafe.ask, one Noul)	412/429	22/39	0	123/123	1	793
Haiku, wording 1	411/429	37/39	22 of 546	120/123	7	512
Haiku, wording 2	850/858 (6 runs)	72/78	2 of 1,092	246/246	11	570
Haiku, wording 3 (shipped)	850/858 (6 runs)	70/78	0 of 1,092	246/246	2	566

• Wording 1 named “a period (‘first week’)” as a NO shape. Haiku then read every claim with a time limit as NO: “the most added song on radio this week”, “first Top 10 single … this week”, “rock ‘n’ roll’s first No. 1 of the year”, and the Joker golden “the biggest October opening weekend in 5 years”. The same lesson as the shape read: a NO example pulls its neighbours.
• Wording 2 narrowed the NO shape to “the sales period of the release itself” and put the near cases on the YES side (“the most added song on radio this week”, “the first No. 1 of the year”, “the biggest October opening in 5 years”). The goldens came back. One borderline claim (“made Billboard chart history, 12 Adult R&B Airplay No. 1s”) was still NO in 2 of 6 runs.
• Wording 3 added one line that states the cost of each answer: “When you are not sure, answer YES: the judge then checks the claim.” Measured, it removed the last true-qualifier NO and cost 2 of 78 false-hit drops. State which error is cheap, and the model leans that way.
• The NO examples are paraphrases (“first-day sales of 40K”, “Best New Artist”, “streams from solo songs only”), not the measured claims, so the set does not grade the model on its own examples.

Rejected, measured: a model on every claim. With the candidates line “none listed; read the whole claim”, Haiku found the 3 list misses but read 504 of 894 misses as YES (“#1 on the US Top 40 radio chart”, “40 billion career streams”), with 61 flips. Every one of those would buy a search and a judge. The list is the right tool for “is there a candidate word at all”; the model is the right tool for “is this candidate a claim”.

The goldens end to end. The read sits in front of the judge and does not change its prompt. Every golden claim (the #4512 table, the #4502 kworb table, the #4468 set, the #4485 ordinals) was YES in all 6 runs and through the shipped method with its cache. So each one reaches check_claim_qualifier with main’s exact input, and its rate is main’s rate.

Worked example: a regex that beat both models (2026-10-18, #4522 site 3)

qualifier_trim._RANK_RE reads one meaning: does the claim rank its subject among other acts by the order they reached a feat (“the 9th female rapper to reach 3 billion streams”)? Its lists exclude a possessive count (“her 13th song”), time (“the 9th week”), release order (“the second single to be released”) and a trip. The owner rule sends such a list to a model, so it was measured.

On 293 real wire tweets with an ordinal before “to” / “with”, the regex was wrong 4 times. Haiku was wrong 65 times in 879 reads and Jev 59, with the regex’s own lists in the task as examples. Both read “This is her 7th song to reach this mark, tying Taylor Swift …” as a ranking: the claim does rank the act among others, but as a tally, not by the order of a feat. A second wording that named the possessive count explicitly (“an ordinal after ‘her’ … is NO”) did not move the number.

Two lessons:

• A narrow regex can be the right tool for a narrow question. This one encodes a grammar (“the Nth to "), not a vocabulary. The models answer the broader question ("is anything ranked here?"), and the examples did not narrow them.
• Check the labels before you trust the gap. One label was wrong (“the 4th ever among all artists” is a ranking the regex misses). The full text of each disputed claim was read before the decision; the gap held.

Worked example: a singular “Top” is a slot, not a list (2026-10-18, CB-02)

market_chart_family sorts a Kalshi board into SLOT_TITLES / FIELD / BINARY / COUNT / OTHER for the chart take gate. The task gave the old title rules’ lead framings as examples, and the rules’ verdict as one input line.

Wording 1 labelled 195 of 198 chart-word series right on 2 runs. Two of its misses mattered. It read “Top Album on Spotify in 2026?” as FIELD, which takes a board the rules gate today out of the gate (a fail-open change on a fail-closed gate). It read “YouTube Charts: Weekly Top Song Global” as FIELD too. Both titles say “Top” with no number, and the FIELD line’s example was “a top 10 list”.

Wording 2 adds one sentence of meaning beside the SLOT_TITLES examples: a singular “Top Song”, “Top Album” or “Top Artist” names the one #1 slot, for a day, a week or a whole year. It also names a sports poll, seed or ranked team under OTHER (wording 1 put the AP poll #1 boards in SLOT_TITLES). Result: 198 of 198 and 186 of 186 random series, 2 runs, 0 flips that change the gate. The new sentence states what the word MEANS; it does not list the two missed titles, so the measurement is not an echo of the fix.

The rule verdict as an input line did not anchor the model: on the three boards the rules miss, the line said “NOT A CHART SLOT” and the model still answered SLOT_TITLES. Without the line (wording 1), the model also missed the YouTube USA weekly board, so the line stays.

Wording 3 (Claude review on #4566). The alert hands the read a LEG snapshot, so the input can carry one leg. With one leg, wording 2 read “#1 on the Billboard 200 chart for the Week of Oct 31” as BINARY on both runs: one title looks like one yes/no question. The fix is two input facts, not a rule: the read now gets the EVENT title (never the leg’s own “Will X rank #1?” question), and the task says “Judge the MARKET line. LEGS shows only some of the market’s legs, sometimes one.” Result, 2 runs each: 198/198 with one leg, 198/198 with the board’s legs, 186/186 random series. The cache key also hashes the exact input now, so a one-leg read never stands in for a board read.

Worked example: a model read that lost to the rule it replaced (2026-10-18, #4522 site 5)

The cinema desk’s tracking pull skips the whole pull when one of its films has opened (#4513). Fixed verb and projection lists decide whether a line or a SOURCES headline reports a result, and an exact title match compares the pull’s films with the weekend chart and TMDB. Under the owner rule of 2026-10-18 this was a site to convert.

The defect. On 70 real SOURCES headlines the lists missed results in real wording: “Rings Up No. 1 $51M+ WW Opening”, “Tanks With $5.5 Million Opening”, “Opens Small”, “Pulls Off a Box Office Shocker”, “a bad weekend at the box office for The Social Reckoning”.

The set. 22 distinct real pulls (36 made with the production query for 6 Fridays), with and without the lane’s opened titles; the 70 headlines as (film, headline) pairs; 54 probes. The script is scripts/dryrun_cinema_desk_score.py --read.

arm	real pulls, lane titles	real pulls, bare	headline pairs	probes	flips
regex	22/22	20/22	527/543	46/54	0
Haiku, wording 1 (one read: list the films and their evidence)	65/66	40/66	1532/1629	144/162	10
Haiku, wording 2 (list every result first, then map it to films)	65/66	56/66	1572/1629	129/162	13
Haiku, wording 3 (code lists the films, the model answers evidence only)	66/66	62/66	1582/1629	154/162	2
Haiku, wording 4 (3 + “an outlook is not a result” + a year example)	66/66 (130/132 over 6 runs)	62/66	1583/1629	153/162	1
Jev (the films of wording 3, one Noul per question)	63/66	63/66	1614/1629	157/162	3

The rule stayed. In production the lane’s opened titles catch every pull the lists miss, so the regex is 22 of 22. Each model adds false skips on real pulls, and a false skip loses a real post. Haiku read “‘Clayface’ Box Office Tracking Points to a Profitable Debut” as a result; Jev read “The Cat in the Hat | $40 million+ opening” as one in every run.

Three lessons:

• Ask the factored question. One read that had to find the films, find the results and join them did the join badly: wording 2 listed “Street Fighter: H4” as a result and then left the film’s evidence empty. When code listed the films (wording 3), the bare score went from 40 and 56 of 66 to 62.
• A rule beside a model example pulls its neighbours. A wording that added “a line that gives a figure with a result verb is a result” moved “previous film Joker opened to $96M” onto Clayface in 3 of 3 runs. It was cut.
• Measure against the system, not against the part. The lists alone were wrong often. The lists plus the opened-title check were never wrong on real traffic. A model that is right more often on the part can still lose on the whole.

Worked example: owner bans read by a model, and a rubric line that was not added (2026-10-18, CB-03)

Six owner bans were decided by regexes alone: no source credit in the line, no confidence disclaimer, no Kalshi-against-HITS comparison, no units ranked against streams, no sum or ratio of board rows, and no previous position on a fall take. Under the owner rule of 2026-10-18 the word lists became the trigger and the model’s input. ClaudeClient.owner_ban gets the line and the matched words, and gives one label per fence. These bans drop a post today, so no answer keeps the drop.

The hit side. 748 distinct real lines from 30 days that one of the regexes matches (493 credit, 26 hedge, 29 row arithmetic, 200 prior position), 2 runs, then 4 runs of the credit fence after its rewording.

fence	hits	rule drops	model keeps	true bans the model cleared	false hits the model cleared
attribution_tag	493	493	492-493	0	0-1 (“Billboard’s list of artists with the most weeks at No. 1”)
confidence_hedge	26	26	26	0	0
row_arithmetic	29	29	7-8	0	21-22 (the band TWICE, Tito Double P, “Don’t Think Twice”, “combined touring, streaming, airplay and social”)
prior_position phrases	200 + 3	203	201	0	2 (“previously announced shows”, “previously unreleased tracks”)

Two words in the first wording moved real credits. “the name of a chart the line reports a place on” let “No. 1 on Mediabase’s US Rhythmic Radio rolling chart” through on 1 of 2 runs, and “Billboard reports” came back as a chart name once. Today the rule drops both. The shipped wording names the only charts the owner has cleared (HITS’ Top 50 and Billboard’s charts), says “Any other publisher’s name is CREDIT, also before the word chart”, and adds “Billboard reports” to the credit examples. The next 4 runs cleared no credit.

The miss side, and why no rubric line shipped. The plan was one hard-fail line per ban in the desk scorer rubrics. First a scratch YES/NO probe read 860 shipped lines that no regex matches (100-170 per ban, from the lanes that run each fence). It flagged 59 lines, and all 59 were wrong when read by hand: “No. 7 on Spotify’s Global Daily chart” as a credit, “per Billboard” on a Billboard chart as a wrong credit, “while its No. 1 debut chance sits at 2%” (one source) as a cross-source comparison. There were 0 real misses. A rubric line would buy nothing measured and would put that false-flag rate into a scorer that drops at 0.0, so it was not added. The note on _IDENTITY_RULE records the same outcome for its record twin (#4275).

Three lessons:

• Name the exception, not the category. “A chart name passes” is a category, and the model extended it to every publisher’s chart. Naming the cleared charts kept the old drops.
• A hit-only prompt is not a miss detector. With no matched words and “when unsure, answer CREDIT”, the hit prompt called nearly every clean line a credit. Each prompt answers the question its caller asks: “what do these words do here”.
• Measure the miss side before adding a rule for it. 0 misses in 860 is the case for no new text.

A count of times is not a ratio (2026-10-24, #3604)

The row_arithmetic regex matches a bare “twice”, so a chart-history line such as “the second album she has sent there twice this year” reaches the model. The OK list named certifications, formats and the HITS name, but no count of times. Haiku read that line as SUM_OR_RATIO 4 of 4 times. The edit adds one OK example: “a count of times something happened ("topped the chart twice", "won it twice")”. The SUM_OR_RATIO list did not change.

Measured with no cache, 4 runs per line, on the 37 distinct 30-day lines the regex matches, 5 paraphrased chart-history lines, and 10 true bans:

set	current OK	new OK
chart-history lines (5)	16/20	20/20
real 30-day lines (37)	116/148 (4 flips)	139/148 (1 flip)
true bans (10)	0/40	0/40

The real lines still read SUM_OR_RATIO are real sums (“top Kanye West and Travis Scott combined”, “LeBron James and Jaylen Brown combined for 33 points”) and one turnover count that flips 1 of 4. In the same 30 days the board lanes logged 0 row_arithmetic drops, so the edit fixes a possible drop, not a measured one. “half the field is new” still reads SUM_OR_RATIO and was left as it is: it is a share of the board’s rows that the model counted itself, which is the risk ungrounded_row_tally exists for.

Worked example: the album metric read, and a title check that stayed code (2026-10-18, CB-12)

The music desk and the music alert cog narrow their dedup history to posts about the same album metric (pure sales, album-equivalent units or streams), and both drop a take that uses another metric’s word. A word table decided both. ClaudeClient.album_metric now gives one cached label per post text, with the table as its examples.

The set. 300 real posts from 30 days that carry a metric word (shipped history and composed lines of the two cogs), 2 runs per wording.

arm	disagreements with the rule	model right	model wrong	effective flips
wording 1	20	15	5	0
wording 2 (shipped)	17	15	1	0

The model is right where the table cannot read the words: “pure-sales” with a hyphen (5 lines), “pure copies”, “equivalent album sales”, and a line that states two figures (“~46.5K first-week units after a 6.2 million-stream opening day”). Wording 1 read “debuts at No. 10 on Billboard’s Top Album Sales chart” as pure sales on 3 lines. Wording 2 adds the chart name as a NONE example, and those 3 came back NONE. Its one wrong line (“The Billboard 200 sales read … ~92K units”, read as two figures) stays in the dedup list, which is the safe side.

The title check stayed code. The plan was a YES/NO read when the exact title fold misses. All 4 fold misses in 2,959 real desk and alert lines were possessives: the fold deletes the apostrophe, so “Silver Sands Marina’s” reads “marinas”. That is a format, so code cuts the possessive “‘s” before the fold. After the fix, the YES/NO read would make 0 calls on real text, so it was not added.

The lesson: read the misses before you choose the tool. “The fold misses some titles” sounds like an open set. All 4 real misses had one exact shape.

Worked example: one route read for the reference router, and a context word that stops a wrong picture (2026-10-19, CB-25)

The /ask reference library picked its source with six intent regexes, in a hand-tuned order. CB-25 replaces them with one cached Haiku JSON read (reference_route): a route label, a chart key and a tables flag. The regex words are the examples in each label line.

Wording 1 read 69 and 66 of 72 labelled questions fully right (the regex chain: 49). Its misses were meaning, not vocabulary: “production credits” for an act read as CREDITS (one song’s credits), “birthplace” read as PERSON (the Wikidata source renders dates and names only), and “how many number one albums” read as ARTICLE once. Wording 2 adds three sentences that say what a label covers (“the songs an act produced or wrote for others are ARTICLE”, “anything else about a person … is ARTICLE”, “counts of them (‘how many number ones’)” under CHART) and one cost line for the flag (“an extra table costs little”). Result: 72/72 routes on both runs. The new sentences state meanings; none of them names a measured question.

The title check needed an input fact, not a rule. The lead-image read (wiki_title_subject) answered YES on “REPO” -> “Repossession” with the subject alone: a fair reading of the word. The caller already knows the subject is a video game. With “(a video game)” after the subject, 17/17 on both runs. Same lesson as CB-02’s event title: give the model the fact the caller holds before you add a rule.

Worked example: “DIFFERENT when you are not sure” refused true pairs, and the model step-up (2026-10-19, CB-28)

The competitor scoreboard’s same-story rule ladder moved to a model read: SAME / DIFFERENT per pair, STORY / OTHER per competitor post. The module docstring’s rules are the definition.

Wording 1 wrote the rules as a checklist and ended with “DIFFERENT = any rule above fails, or you are not sure.” On 30 days of real pairs, Haiku refused true pairs that carried one extra fact: an ordinal on the same milestone, a record claim beside the figure, a release question beside the release. It was worse than the rules on about 40 pairs. The cost line pulled the answer the wrong way: neither error is cheap here, so the line had no business in the task.

Wording 2 names the main fact first (the same act and work, the same chart or milestone, figures that agree within 2%), then the details that do not change the story, then the facts that make two stories (two charts, a different figure, a forecast against the result, a release against its chart entry). The examples are paraphrases, not measured posts. Haiku went from 72 to 57 wrong; it still refused about 27 true pairs the rules got right.

The step-up was measured, not assumed. The owner rule allows Sonnet only when Haiku is worse on the same inputs. Same prompt, same 2,009 pairs: Sonnet 16-17 wrong, 7-8 worse than the rules, 8 flips. The pair read ships on Sonnet; the post read, where Haiku made 6 errors against the rules’ 56, stays on Haiku.

Worked example: a DISTINCT line that pulled every subtitle, and a list that stays as a block (2026-10-19, CB-17)

album_variant_relation decides whether a longer album title on a kworb page is the same record as the shorter one it starts with. CB-17 sends every pair to it, not only the pairs the word list cannot place. On 311 hand-labelled real pairs the old prompt was wrong 44 times, against the word list’s 13.

The cause was one clause. The DISTINCT line ended “or a new album whose title adds words to the other”. Every deluxe edition with its own subtitle adds words, so Haiku read “CALL ME IF YOU GET LOST: The Estate Sale”, “Unreal Unearth: Unaired” and “MEGAN: ACT II” as new albums. The same lesson as the qualifier read: a NO example pulls its neighbours.

Before:

VARIANT = the same album in another edition or version: deluxe, expanded, a bonus or
special edition, a named edition, a chopped or slowed version.
DISTINCT = a different album: a sequel, a numbered volume or part, a re-recording, or a
new album whose title adds words to the other.

After:

VARIANT = the same album released again in another form: a deluxe, expanded, bonus,
special, anniversary or tour edition, often named with a subtitle after a colon or a dash;
a remaster or a new mix; an edition with extra songs; instrumentals, a cappellas, remixes,
commentary, chopped, slowed or sped-up versions; a karaoke version of that album.
DISTINCT = a different album: a sequel with its own number ("SAVAGE MODE II"); a
re-recording such as "(Taylor's Version)"; a live album recorded at a concert; a
soundtrack; a compilation; or another album whose title starts with the same words.
UNSURE = you do not know these records well enough to tell. A wrong answer states a false
album count, and UNSURE only leaves the count out, so answer UNSURE when you are not sure.

Result: 44 wrong to 18. Two more lessons:

• The rule’s verdict as an input line did nothing here (41 wrong without it, 44 with it), so it is not in the prompt. The rule’s words are in the label lines instead.
• A model that is better overall can still lose a class the rule never loses. The new prompt calls numbered sequels variants (“A Love Letter To You 5”, “Artist 2.0”). So the shipped rule keeps the list as a block: a pair the list places counts only when the model agrees. Result: 12 wrong and 18 withheld, against 36 wrong before. A withheld count is safe; a wrong one is a false claim.

Worked example: a “Top 3” example the judge copied (2026-10-18)

The Taylor Swift card heroed “Top 2”. Owner: “stop doing Top N, and just always name the position like #N”. The wire was Billboard’s “Artists With the Most Weeks at No. 1: Full List” post, which states no position. The verify judge (Haiku) wrote the figure.

The cause was an example in the prompt, not a missing rule. Both verify_music_claim and classify_music_news carried “#1, #2 and #3 is ‘Top 3’”. It was the only rank example in the judge’s figure line, so the judge copied its SHAPE onto a ranked-list stat. The sentence was CUT from both prompts; collapse_rank_run turns a list of positions into “#1”, the best current position, in code (owner steer 2026-10-19: “i want the current position … always”). Measured on the real judge with live research, n = 10 per arm:

Prompt	Taylor research: “Top N” figures	Drake “#1, #2, #3” post: figure
before	3 of 10 (“Top 2 all-time” x2, “Top 3” x1)	“Top 3” 10 of 10
after the cut	0 of 10 (“100 weeks at No. 1” 10 of 10)	“#1, #2, #3” 10 of 10

The classifier, n = 6 per arm on four posts: the Drake lockout post went from “Top 3” 6 of 6 to “#1” 6 of 6, with the subject “ICEMAN - Drake” and the top-3 fact in the claim. Its other three posts did not change (“#2” for a #2-#4 run, “top 10” for a band-only post, “#4” for a stated position).

The new Haiku call (name_chart_positions) needed one added clause. A band that is still the figure goes to Haiku with the post and the research. The first prompt asked for the subject’s “chart POSITION”, and the judge read a rank on a ranked list as something else (“states the rank as ‘second’ rather than as a chart position number”). Taylor came back #2 in 2 of 5. Rewording to “its rank number” gave 5 of 8. Adding “A rank on a ranked list (the most weeks at No. 1, the best-selling artists) is a position too.” gave 10 of

1. The goldens held at 6 of 6 each after the clause: a band-only post with band-only research gave [], and Tinashe’s “5 of the top 8” gave []. Rubber Soul gave #4 and Drake [1, 2, 3] (hero “#1”), 6 of 6 each. One case stayed noisy: a radio story whose research named only “No. 20 apparently” on another chart gave #17 in 2 of 6. So code checks that the number is written as a position in the text the model read (mn.position_grounded).

A finding that did not reproduce gets no rule (Codex review on #4560). The card heroes the BEST position the subject holds, so a past peak in the resolver’s answer would win. The review asked for a “never a peak or an earlier week” rule. Measured on the current prompt, n = 8 each: research “re-entered at No. 4 … peaked at No. 1 for six weeks in 1966” gave [4] 8 of 8, and “fell from No. 8 last week to No. 16 this week” gave [16] 8 of 8. No rule was added; both stay as goldens for the next change to this prompt.

Worked example: a title the model read as words said to the artist (2026-10-23, #4510)

The gated eval classify_music_news_subject_title went red after #4560 (0.83 against the 0.85 floor). Its golden is the live chartdata line “Billboard 200: #2(+3) @oliviarodrigo, you seem pretty sad for a girl so in love 67,000 …”. On main the classifier answered “GUTS - Olivia Rodrigo” 10 of 10.

The first suspect was not the cause. #4560 cut the “top BLOCK” sentences from the figure rule. Restoring them gave 7 of 10 grounded, but every pass was the bare artist (“Olivia Rodrigo”), which mn.title_grounded_in_wire counts as grounded. The old prompt never copied the title. Each cut sentence restored alone gave 0 of 10. So the cut text did not hold the title rule up; its effect was an unstable interaction, and restoring it would bring back the “Top 3” example the owner banned. Ablation, live Haiku, n = 10 per arm:

Arm	Answers	Grounded
main	GUTS 10	0/10
restore all #4560-cut text	artist only 7, GUTS 3	7/10
restore one cut sentence (each of the two)	GUTS 10	0/10
cut “never a title you know from the act’s catalog”	GUTS 10	0/10
title rule without the catalog words	GUTS 10	0/10
“in a chart line it is the words after the artist” (no example)	GUTS 8, artist 2	2/10
“in a chart line (‘Hot 100: #7(-2) @artist, title’) it is the words after the artist”	title 10	10/10
“a chart line (‘Hot 100: #7(-2) @artist, title’) prints it right after the artist”	title 10	10/10

The cause was a missing fact in the title sentence. The model read “@oliviarodrigo, you seem pretty sad …” as words said TO the artist. The sentence told it a title can read like a sentence, but not WHERE the title is. The fix rewords that sentence to say where; it adds no counter-rule. The example line is a generic Hot 100 line, not the fixture’s shape, and the version without an example did not work.

The wording “the words after the artist” pulled a list into the subject. On a post naming three albums (“holds #2, #3 and #4 … with Midnights, Lover and folklore”), it gave “Midnights, Lover and folklore - Taylor Swift” 3 of 10. “prints it right after the artist” gave one title on 37 of 40 runs (main: 35 of 35). That residual is accepted: the list post is synthetic, and the title is still grounded in the wire.

After the fix the eval scored 1.00 on 3 of 3 runs. The other five classify_music_news evals passed 3 of 3, and the #4560 figure posts did not change (Drake lockout “#1”, a #2-#4 run “#2”, a band-only post “top 10”, a stated position “#4”, n = 10 each).

Worked example: give the judge the fact the compose got (2026-10-24, #4719)

#4713 gave market_drop’s open report lock a STATUS line: the market has not settled, so say who is on track. The compose obeyed and wrote “Animals is on track to top Netflix’s global movie chart this week.” The score check (drop_score(report=True)) did not get the line. Its rubric says the market is SETTLED, so it read “on track” as a live betting line. It scored the real lines 0.14 and 0.22 for ODDS FRAMING and 0.45 as a hedge.

The cause was a missing input, not a wrong rule. Two fixes were measured on the real Haiku judge, the live Animals board, n=5 per line:

arm	change	3 “on track” lines pass	3 settled-wording lines pass
base	none	7/15 (one line 0/5)	15/15
status	STATUS line appended to the ground truth	15/15	15/15
reword	an open_lock premise in the rubric	15/15	6/15

The rubric reword passed the new lines but broke the old ones. It invented a “quotes the settled %” fail on “Animals is this week’s Top Global Netflix Movie.” The shipped fix is the status arm: the judge reads the same fact the compose read, and no rubric text changes. When a compose gets a new fact, give its judge the same fact before you reword the judge.

Worked example: a borrowed angle and a borrowed premise (2026-10-24, #4391)

bet_value posts a VALUE call: the book has a side at +300, the markets have it at 42%, so the book looks slow. It borrowed two texts written for other posts, and each one fought it. Measured on main, gpt-5.6-terra compose, Haiku gate, 8 games, 2 runs each: 3 of 8 games passed both runs, and only 2 of 8 kept the book’s odds.

• The angle. It composed on race_moved: “the favored side + its %, then STOP”. The model obeyed and posted “Inter Milan vs Napoli: Napoli at 42%.” The value call was gone. A value angle that says what this post is fixed it: 8 of 8 takes kept the odds.
• The judge’s premise. The default drop_score rubric opens “what the prediction markets reveal … led with a real market number”. Haiku then did the odds math itself (“+300 implies roughly 25%”) and failed the clean value take as UNGROUNDED. Gate-only on the 8 real takes: 14 of 24 passed. A reading line in the ground truth gave 16 of 24. value=True, which rewrites only the premise sentence, gave 40 of 40.

A fail clause in the same premise did not hold. “A number the post works out from the odds is still UNGROUNDED” passed the clean takes 22 of 24, because its example “+300 is 25%” used the same odds as one fixture and the judge copied it into its reason. Without the example, the clause passed clean takes 40 of 40 but let the bad takes through (wrong odds 2 of 3, a point gap 3 of 3). One small judge did not do both jobs. The shipped shape splits them: the premise says the numbers are meant to disagree, and the code check (ungrounded_numbers) drops any figure the blob never gave. When a premise has to tell the judge to stop checking something, check that thing in code.