a discord bot for the tootsies server. ask, recap, discuss, ship features by typing.
CLAUDE.md carries the binding rules (the operability bar, the field-budget core, the
ops-monitor summary); this file is the full narrative those rules compress – the
complete field-budget history + tooling, and the ops monitor’s findings catalog.
The DETAILED event ledger (the full table) lives in this file; utils/events.py’s
module docstring carries the compressed quick index – keep both updated.
Every metric-worthy thing emits a JSON event line via utils.events.emit(kind, **fields).
Each line is prefixed with the literal EVENT so Railway log queries can isolate
dashboard data from operational logs.
Existing event kinds – the DETAILED ledger (the compressed quick index lives in utils/events.py’s module docstring; keep BOTH updated when adding an event):
| event | source | fields |
|—|—|—|
| command | utils/metrics.py (@track_command) | cmd, user_id, guild_id, duration_ms, ok, error |
| claude_api | claude_client.py (_call) | model, purpose, input_tokens, output_tokens, duration_ms, stop_reason, ok. On failure: error (the exception CLASS name) plus detail.api_error — the API’s own message, truncated to 300 chars (#2174), read with parse_json(detail)['api_error']. The class name alone is not diagnostic: a robots.txt refusal, an image the API could not download, a malformed tool schema and a blown context window all arrive as BadRequestError, so 22 such events over 30 days said a call failed and nothing about why. A 400 is deterministic and caller-caused — the body is the answer. Parity with openai_api, which already stored its response body, and it was that stored body that made #2171 solvable in minutes. On success, detail.cache_creation_input_tokens / detail.cache_read_input_tokens (#2659) — the Anthropic prompt-cache write/read token counts for this turn, read with parse_json(detail)['cache_read_input_tokens']. input_tokens on its own is the UNCACHED remainder ONLY (it excludes both), so before this pair shipped there was no way to tell a cache hit from a miss, or to add up what a call actually cost — the gap a low org-wide cache-hit-rate report (Console billing alert, 2026-08) surfaced: the mechanism (cache_control: ephemeral on the persona + per-surface system blocks, see ARCHITECTURE.md) was already live and, verified live against the running persona prompt, DOES cache warm for the high-volume Sonnet/Haiku surfaces (cache_read_input_tokens > 5000 on the very first probe call) — the org just had no per-call visibility into it. The low-volume Opus /ask surface (~3 calls/day) is a genuine, expected cold-cache surface: its call gaps run ~8h, past even the 1-hour TTL, so there is no TTL choice that would warm it — accept the miss rather than pay a pre-warm write nobody would read back. scripts/axiom_setup.py’s cache_hit_stat/cache_hit_purpose panels and the ops-monitor’s per-provider “LLM spend by provider” line (cache hit N% (reads / writes)) both read this pair; the cost panels/_spend_cost price a cache write at ~1.25x and a read at ~0.1x the model’s input rate, so the estimated spend actually reflects a cache hit’s savings now instead of silently excluding cached tokens from the sum entirely. On success, detail.image_retry (#backpage, 2026-08-30) – the API rejected one of the attached images, so _call dropped EVERY image and retried on the text alone. Read the rate per surface with ['tootsies'] | where event=='claude_api' | summarize calls=count(), retried=countif(tostring(parse_json(detail)['image_retry'])=='true') by purpose | extend pct=round(100.0*retried/calls,1). Match on the VALUE, never on the substring — emit() keeps the null keys when it stringifies detail, so every claude_api row literally contains the text "image_retry":null and a detail contains 'image_retry' filter matches 100% of calls (measured 2026-09-03: 45,590 of 45,590 over 7 days, against a true 476). The same trap applies to every marker in this row. Measured correctly, curator_pick retried 475/2454 calls (19.4%) over the 7 days to 2026-09-03. It sat near 10% of curator_pick calls for months, climbed past 50% in August 2026, and nobody saw it, because the retry still reports ok=true. That is the fail-open trap: the vision fit-judge kept answering about images it could not see. A caller whose TASK is the images now passes images_required=True, which turns the retry off – the call FAILS instead, and the failure carries detail.api_error naming the image. So watch image_retry as a degradation rate, not as noise. Since #2732 every vision JUDGE sets it (curator_pick, fighter_likeness, media_coherence, media_text_evidence, gif_pick, subject_image_pick, subject_image_confirm; since #4145 also clip_coherence, tour_admat, wire_identity, image_match), so those purposes never show image_retry. curate_route does not set it yet (named on #4145); an image they lose shows as an ok=false call whose api_error names the file. The LLM board’s vision_image_loss panel counts both shapes by purpose (retried and refused). Full account: ARCHITECTURE.md -> curator.py. detail.thinking_exhausted / detail.thinking_recovery (2026-09-03) – a thinking-enabled turn that stopped at max_tokens with NO text (every token went to thinking blocks, which are dropped) is stamped thinking_exhausted=true, and the one low-effort retry of that turn is stamped thinking_recovery=true. Before the retry existed the surface shipped its empty-answer fallback (“lost my train of thought”) and answer_length read as a normal 45-char reply, so the miss was invisible. Query ['tootsies'] | where event=='claude_api' | where tostring(parse_json(detail)['thinking_exhausted'])=='true' | summarize count() by purpose, model — match the VALUE, not the substring (see the image_retry note above: a detail contains filter matches every row, because the null keys survive into the stringified detail); a recovery turn that ALSO stops at max_tokens is the still-empty case — that is what the ops monitor’s reasoning_exhausted finding flags (#2934), and the llm_exhausted dashboard panel graphs both providers’ markers. |
| openai_api | utils/openai_chat.py (complete/create_response) | model, purpose, duration_ms, ok; on success: input_tokens, output_tokens, stop_reason, response_chars, response_preview (Chat Completions); Responses round-trips carry responses=True; on failure: error (HTTP <status>/no_text_in_response/<exc>), detail, category (#973 — the failover class: provider/quota/auth/request/malformed; provider+quota fail over to Claude, the rest surface; category=quota = OpenAI out of credits, a 429/insufficient_quota NOT retried), detail — one OpenAI call, the optional OpenAI TEXT backend (#openapi-parity), the provider twin of claude_api for a surface routed to a GPT model (cost-spread / Claude-credit failover). Two paths: Chat Completions (tool-free) + the Responses API (tools/web_search, one event per round-trip). Ops-monitor keys these openai:<purpose> (parity with claude_api’s per-purpose split) so each GPT-routed surface’s cost/latency stays separately visible. On success, detail.reasoning_exhausted / detail.reasoning_recovery (#2869) — the openai_api twin of claude_api’s thinking_exhausted pair. A reasoning model (the gpt-5 family) spends its thinking out of the SAME output budget as the visible message, so a turn that spends it all comes back with no text — status="incomplete" on the Responses path, finish_reason="length" on Chat Completions — and the surface ships its fallback line. Such a turn is stamped reasoning_exhausted=true, and the one lower-effort retry of it is stamped reasoning_recovery=true. Query ['tootsies'] | where event=='openai_api' | where tostring(parse_json(detail)['reasoning_exhausted'])=='true' | summarize count() by purpose, model — match the VALUE, not the substring (see the image_retry note in the claude_api row); a recovery turn that is ALSO stamped exhausted is the still-empty case — the ops monitor’s reasoning_exhausted finding flags it (#2934) — and a sustained rate on one purpose says that surface’s budget floor is too low for what it asks the model to do |
| typesafe_api | utils/typesafe_client.py (TypeSafe._ask, via @instrument) | purpose, model (the pinned jev-1.13.0, or the versioned id that answered), ok, duration_ms, count (questions asked), api=typesafe; on success: input_tokens; on failure: error (HTTP <status> / breaker_open / no_questions / the exception class) and http_status for an API answer. A caller’s @scoped_surface adds surface + path (<surface>.typesafe_api). One event per ask — the final outcome after up to 3 attempts, not one per attempt (#4310). ask fails OPEN (returns None, the caller keeps its current judge), so this ok-rate is the only signal of a dead key or an outage: ops-monitored as the typesafe health line, p99 ceiling 2 s, and the Axiom monitor “TypeSafe Jev calls failing”. Jev spend lands on the typesafe provider line at $0.042/1M input tokens |
| typesafe_shadow | utils/typesafe_client.py (emit_shadow) | purpose, ok (= Jev gave the same verdict as the current judge); current + jev (the two verdict LABELS, cut to 40 chars, never message text) ride detail — read them with parse_json(detail)['current']. Emitted by a surface that compares Jev’s verdict with its old judge’s (#4310; first user #4311), and ONLY when Jev answered: a None from ask emits no shadow row, so agreement measures real answers only (the failure is already a typesafe_api ok=False row). REPORT-ONLY (owner direction): the ops monitor renders a typesafe_shadow:<purpose> agreement line and raises NO finding, and no Axiom monitor reads it, because disagreeing with an old grader is expected when that grader is the bug being fixed. /guess credit (CB-15): purpose=guess_credit, one row per artist or guest credit the deterministic rule nominated and Jev answered; current = the rule’s credit and jev = the credit after Jev, each both / artist / guest / none. Panel guess_credit_shadow (jev_dropped = chat the rule credited and Jev did not). Mention shadow (#4406): purpose=abuse_gate (labels abuse / ok: the Haiku gate vs Jev’s Noul) on every mention that reaches the abuse gate, and purpose=mention_route (labels ask, icebreaker, music, recap:<1h\|1d\|today>: the Sonnet router vs Jev’s Choice; discourse reads ask) only where the router ran. Both carry surface=ask. The route comparison is judge against judge: it uses the router’s raw verdict, before the cog’s other-channel recap -> ask override (_names_other_channel). That override is deterministic and would apply to a Jev verdict in the same way, so it is left out of both sides. A recap on a mention that pings another channel is therefore not what shipped. The one Jev request is typesafe_api purpose=mention_shadow; a 3 s timeout is an error source=mention_shadow, recoverable, with surface=ask. Shadow only: the reply never waits for it Wire admission (CB-09): purpose=wire_admission, one row per wire post the newsroom’s cue lists drop (mn.cue_dropped: no numbers cue, or a culture post with no domain cue) that Jev answered; current is always drop, jev is story (P >= 0.5 that the post reports a music number) or drop, and detail.external_id is the post id, so the shadow week can fetch and label the story rows. Each post is asked once per process, at most 150 a slot, in a background task with an 8 s timeout per call and one 4-call cap shared by every pass; the typesafe_api request has the same purpose and surface=music_news. A timeout or an unexpected error is an error source=wire_admission_shadow, recoverable, surface=music_news. Every post that gets no shadow row (a timeout, no answer, an answer without the key, an error, the key gone) leaves the memo, so a later slot asks again. Panel wire_admission_shadow. It changes nothing that posts |
| curator_frame_fetch | cogs/curator.py (localize_vision, called by the curator, the artist curator and the curate feed) | count (images that downloaded to inline base64), ok (at least one downloaded), duration_ms; dropped (# that did not download) folds into detail; surface=curate when the curate feed called it. Since #4287 the vision judge gets NO image URL: every candidate frame and channel example is downloaded with an 8s deadline per image and scaled to a 1568px long edge, and a miss drops only that image. Since #4303 dropped also counts an image the download REFUSED: a body over 5MB, a header over 50M pixels, a non-http(s) URL, or a host with a non-public address. Before, the API fetched every host outside a TikTok/Instagram allowlist and failed the whole call on a robots.txt refusal or a slow host (69 + 6 of 169 curator_pick calls in the 7 days to 2026-10-16). Its OWN kind, not curator_fetch, so a CDN block cannot corrupt the twitterio signal. Since #4298 retried and recovered (both in detail) count the Discord embed proxy fallback: an embed image whose url gave no image (a kkinstagram url redirects to HTML) and had a proxy_url is retried from the proxy; retried counts those tries and recovered the ones that gave an image. A url that works never touches the proxy. Panel: “Vision-frame fetch health” (proxy_tried, proxy_ok). Ops-monitored as integration source_cdn, and gated by a 10s p99 ceiling. |
| openai_vision_fetch | claude_client.py (_localize_openai_vision) | purpose, count (images that downloaded to inline base64), ok (none dropped); dropped (# we couldn’t download) folds into detail — OpenAI’s Responses API can’t server-side-fetch an image URL (#1365), so the bot downloads it and passes bytes. An image that WON’T download is dropped (#2171): passing its URL made OpenAI reject the whole call with a 400 (Error while downloading file), and a 400 is category=request, so the router re-raises instead of failing over to Claude and the surface posts its canned error quip — one unreachable image killed the whole answer (the 2026-08-08 ask_mention outage: a reply to a message whose image had expired, host 401). Dropping costs that one image and keeps the reply; the paired [image N, posted by …] label block stays, so the model is not told an image is present when it isn’t. The drop raises NO exception (just a thinner prompt), so this ok-rate is the only signal our IP stopped reaching an image host — ops-monitored as integration vision_cdn. retried / recovered (in detail, #4298) count the Discord embed proxy fallback, as on curator_frame_fetch. A low steady rate is NORMAL (an expired Discord attachment URL); only a blackout flags |
| provider_fallback | claude_client.py (_call) | purpose, requested (the model that failed), fell_back_to, reason — one provider failed and _call failed over to the OTHER (#openapi-parity, now BIDIRECTIONAL). FORWARD (a GPT surface → Claude): fell_back_to = the GPT’s Claude sibling (gpt-6-sol→Opus, gpt-5.6-terra→Haiku; an unmapped id→the frontier Opus), reason = the failure CATEGORY (#973: provider/quota/unprovisioned — the FALLOVER classes; an auth/request/malformed OpenAI failure is NOT failed over, it re-raises so the surface’s own canned fallback handles it, the mirror of the reverse direction re-raising a Claude 400/401) — the graceful degradation that makes gpt safe as a per-surface menu choice. REVERSE (a Claude surface → OpenAI, #973): fell_back_to = the Claude’s OpenAI sibling (Opus→gpt-6-sol, Sonnet/Haiku→gpt-5.6-terra; an unmapped id → the frontier gpt-6-sol), reason=credit_exhausted (the Anthropic balance ran dry — a 400 naming the credit balance, permanent until topped up) or claude_unavailable (a persistent 529/5xx/429/timeout that survived the retry loop) — credit-exhaustion failover, so the bot keeps talking instead of going dark on every surface at once. Each direction calls the OTHER provider’s method DIRECTLY (never back through the router), so the two can’t ping-pong. A rising FORWARD rate = OpenAI flaking; a rising REVERSE rate = Anthropic out of credits / overloaded. (Vision is served on the Responses path now, so it no longer triggers a fallback by itself.) |
| order_state | cogs/order.py | order_id, issue_number, guild_id, user_id, from, to, via (reconcile, or expired when the sweep burns a PREPPING order with no PR after ORDER_STALE_DAYS, #4684) |
| order_served_ping | cogs/order.py | guild_id, user_id (the reporter), order_id, issue_number, delivered, reason (None on delivered; no_guild/no_channel/no_perms/bot_off/send_failed) — when a reconcile flips an order to SERVED, Toots pinged the original reporter back in the channel they filed it from (“completing the circle”). Fires once per order (guarded by orders.served_pinged_at); fail-open, honors the master kill switch (bot_off leaves it pending so it lands once she’s back on) |
| rate_limit_hit | utils/rate_limits.py | scope, command, user_id, guild_id, count, cap |
| deploy_event | bot.py | kind (boot/shutdown), guilds |
| guild_join | bot.py (on_guild_join) | guild_id, ok, + folded detail (name, members, owner_id, inviter_id) — Toots was ADDED to a server. Captures the guild NAME + who added her (the facts that were unrecoverable once she’d left an unknown server), AND pages the bot MASTER in the MASTER guild’s #bot-logs (utils.permissions.MASTER_GUILD_ID) so an unrecognized add is surfaced the moment it happens, not discovered weeks later from a stale DB row. Fully fail-open. tier=games_only (2026-10-18) when the server never ran /menu: that server is in GAMES-ONLY mode (only the arcade runs, cogs/arcade.py gate_open), and the #bot-logs page adds a line that says so and names the two dashboard panels. tier is absent on a configured server, and absent when the is_configured read fails (the page still goes out) |
| guild_left | bot.py (on_guild_remove) | guild_id, ok, tier, + folded detail (name, members, owner_id) — the mirror of guild_join: Toots was REMOVED from a server. A durable record of WHEN she left + a master #bot-logs page. tier=games_only and a page line when the server never ran /menu |
| command_refused | bot.py (_GatedTree.interaction_check) | guild_id, command, reason (arcade_only), tier (games_only) — a slash command refused in an ARCADE-ONLY server (2026-10-18, owner: “other than those two the only thing that should work is /fight”). Only Tootsie’s and the master guild get the whole bot (utils.permissions.full_bot_allowed); every other server is synced /fight alone (_sync_guild), and this event fires when a client still has another command cached and runs it, /menu included. /fight never emits it. Counted on the games_only_servers panel as refused_cmds. No monitor: a refusal is the intended outcome |
| error | cogs/* + bot.py error handler | source (e.g. ask, order_preflight, undo), error (exception class), guild_id, user_id, optional context, optional status (the HTTP status where the call site stamps it; source=deezer does since #4756, so a 403 IP throttle and a 5xx outage are separable), traceback (last 3 frames), + folded detail.error_message — the exception’s own TEXT, capped at 300 chars, and OFF unless the call site passes include_message=True (#2816). The default is a data-minimization rule: emit_error is the catch-all around provider / HTTP / Discord / database calls, so str(exc) is UNCONTROLLED input — an asyncpg unique-violation prints the offending key VALUE, an aiohttp error prints the request URL with its query string, an LLM SDK error can print a response body — and none of that may reach a durable 30-day store. Opt in ONLY where the text is ours: a constructed exception, written to describe a fail-open miss, which has no traceback either, so its message is the only diagnosis it has. Seven such sites (billboard, pollstar_charts ×2, industry_feeds ×2, link_enrich, music_desk) are opted in. Query with parse_json(detail)['error_message'] Two sources added 2026-09-28 after a silent week-long newsroom blackout: <surface>_scheduled with recoverable=false is a ScheduledPoster slot whose compose RAISED, so the whole slot was skipped (it was a log line only before, and the music newsroom crashed there every slot from 2026-09-21 to 2026-09-28 with no finding); music_news_compose is one newsroom story whose compose raised, now contained to that story so the rest of the slot still posts. Either one repeating every hour means a surface is posting nothing. music_news_feed_watch (2026-09-28, retired by #3834) was one pass of the newsroom’s own 2-minute feed watcher loop failing. That loop is gone: the RSS feeds are now the second source of the newsroom’s EventPoster lane, so a failed pass of that lane (both sources) emits music_news_breaking (the spine’s <surface>_breaking source, recoverable=true), and one feed read failing emits music_news_feed. The same source every pass means the lane is dead and first-party stories wait for the hourly slot again. music_desk_source_watch (2026-09-28, recoverable=true) is one pass of the music desk’s 2-minute source watcher failing; the next pass retries. The same source every pass means the watcher is dead and every desk story waits for the hourly slot again. |
| recap | cogs/recap.py (produce_recap) | guild_id, period (1h|1d|today), count (messages used), decision (ceiling|rate_limit|size|full) — one recap built, with its coverage. A non-full decision means the recap covers only the most-recent slice of the window, and the card’s header (and the @-mention reply) state the REAL span (“the last ~3 hours”) instead of the period label. The cause is recorded as ONE reason so the dashboard reads the ceiling signal alone: ceiling = the window held more than the per-period message ceiling (the fetch asks for one message past the ceiling, so a cap is only recorded once a further in-window message actually exists — never inferred from how close the span looks); rate_limit = a global Discord 429 cut the history walk short (recent_messages’ interrupted signal); size = the rendered blob overflowed the token budget (_MAX_BLOB_TOKENS, which drops the oldest messages — and clips a single oversized message — so a transcript-heavy channel degrades instead of hard-failing on a context-length error). Only a high ceiling rate on 1d/today says the ceiling is too low (the signal behind the “/recap today only recapped the last hour” regression fix); a size/rate_limit cap is a different problem. Dashboard: the recap_cov panel on the ask/conversation board breaks the rate out by cause. Not an error (a capped recap is honest, just partial), so no monitor. |
| recap_deflected | cogs/recap.py | guild_id, user_id, period, channel_id, channel_name, reason (no_permission/no_messages), can_read_history, total_messages |
| discourse_fallback | cogs/discourse.py | guild_id, user_id, category, source_count, recent_topic_count, reason (claude_returned_empty = the model returned the EMPTY skip sentinel, a deliberate decline, not an error; claude_returned_blank = no text at all; self_correction = the shape guard dropped a redrafted line). What follows depends on the caller: a manual /discourse gets the quip, a scheduled slot gets an icebreaker, the X-only lane skips. The #bot-logs line names the same reason and destination (it used to say empty_claude_response + quip on every exit, 2026-09-20). Retired 2026-10-15 (#1871): the discourse loop is dormant, so no new rows after that date. |
| wire_read | utils/wire_sources.py (gather_wire), called by cogs/discourse.py and cogs/music_desk.py | surface (discourse | music | music_desk), guild_id, category, count (handles asked for), kept (posts returned), ok, duration_ms — the direct read of the CODE-OWNED wire lists that a compose UNIONS with the guild’s feed channels (#wire), so a server with no MEE6 mirror still has linkable material. count is the breadth the surface allowed (music_wire_accounts for the music surfaces, 0 = the direct read is off for that guild and no event fires at all; retired discourse reads the code default wire_sources.DEFAULT_PER_LIST, its knob was deleted in #4175). The music DROP deliberately does NOT read this (PR #2202): a live A/B showed news material pulls the compose onto the news subject and it fabricates a track, so surface never carries music. The wire desks’ breaking EVENT poll (cogs/event_poster.py, #news-breaking-speed) also emits this kind with surface = the desk and category = breaking — the same fail-open read of the head wire handles, on the aligned 15-minute poll. The fail-open tripwire is kept: the fetch never raises (a dead handle, an exhausted twitterapi.io balance, and a missed deadline all return []), so a kept=0 with count>0 across many composes is the SILENT degradation to watch — the surface keeps posting off the feed channels alone and nothing errors. duration_ms covers the whole concurrent fan-out, not one handle — and is near-zero on a warm cache, since the desks’ own fetches serve most composes. The kind is surface-neutral on purpose: one shared integration read by three surfaces stamps surface rather than splitting into per-surface kinds, so one panel and one health branch cover all of them. Ops-monitor health: rolls into misc_health["wire_read_<surface>"] — keyed PER SURFACE so a healthy discourse cannot mask a music read that has gone dark — and scored on kept>0 rather than on ok for exactly the reason above. |
| discourse_scored | cogs/discourse.py | guild_id, channel_id, score, reason, must_post, category, user_id, post_preview; on category="trending" also (#1272 silent-degradation tripwires): shipped (bool — the trending-clip reaction cleared its high solo bar and won the slot vs fell through to a news take), platform (the clip she reacted to), framed (# pool clips that carried a vision cover frame — a framed=0 while TikTok is in the pool means the frame plumbing broke, e.g. TikTok changed its cover-URL shape or Anthropic stopped fetching tiktokcdn, so she’s back to reacting caption-blind), pool (the balanced platform mix, e.g. instagram:2,tiktok:2,youtube:2, confirming the per-platform balance holds). #trending-visibility: it ALSO fires on the PRE-score DECLINES (shipped=False, score=0.0, reason=no_clips|model_declined|error) so EVERY trending attempt is countable — the ops-monitor renders a “Surface decisions” line (news vs trending shipped + the decline breakdown) and flags trending_dark when trending is attempted repeatedly but never ships on a fixable reason (error = the GPT-vision Responses 400 that took it dark; no_clips = the gather finds nothing on-theme; model_declined is by-design restraint, never flagged). The GPT-vision failure now degrades caption-only in react_to_trending (drop the frames + retry once) rather than falling through to a news take every slot. Retired 2026-10-15 (#1871): the discourse loop is dormant, so no new rows after that date. |
| discourse_dedup | cogs/discourse.py | guild_id, channel_id, decision (similarity_gate), signal (same_link/text_similarity/shared_run/content_overlap), post_preview Retired 2026-10-15 (#1871): the discourse loop is dormant, so no new rows after that date. |
| music_desk_scored phase=projection | cogs/music_desk.py (_compose_line, #3913, 2026-10-25) | no new event: the sales-projection lane’s score, its pre-judge drops (ungrounded_number, cut_qualifier, the ranking guard) and its claim_arbiter rows now carry phase=projection. Before, _compose_line set the phase from the reading (reveal if settled, else tracking), and a projection reading is always live, so every projection score read phase=tracking: 4,349 of the 4,845 tracking scores in the 30 days to 2026-10-25 were projection lines. music_desk_posted and post_dedup already said projection. The two shape_reject rows that _compose_line emits (reason=metric_word:*, reason=release_title) keep phase=shape_reject and now name the lane in detail.lane. Query: where event == 'music_desk_scored' and phase == 'projection'. No dashboard panel or monitor filters phase == 'tracking'; the ops monitor’s per-lane self-gate check now shows music_desk/projection as its own lane. |
| music_desk_scored phase=chart genre | cogs/music_desk.py (2026-09-30) | no new event: a chart card that the GENRE DEBUT rule let through (a Billboard 200 debut that also debuts top 15 on a genre album chart) carries genre = <genre chart>#<rank> (folded into detail). Query: where event == 'music_desk_scored' and phase == 'chart' and isnotnull(parse_json(detail)['genre']). See docs/BILLBOARD_ALERTS.md. |
| music_desk_scored (any phase, shipped=false) – the COOLDOWN NOTE | cogs/music_desk.py (2026-09-06) | no new event: a shipped=false row from _compose_board_unit, _milestone_candidate or _compose_record_unit now also stamps fail:<dedup key> in music_desk_events, and every lane skips that key for MUSIC_DESK_FAIL_COOLDOWN_HOURS (12). So a pick that fails reads as at most ~2 rows a day instead of one per slot (before: one radio board 165 rows in 7 days, one record card 61, one chart-peak card 73). Read a lane’s failure rate as dcount(matchup) alongside count() – the same failing pick no longer inflates the count. A skip emits nothing (it is derived state, and a per-slot skip row would re-inflate the very count this fixes). reason=stale_score, phase=reconciliation, shipped=false, score=0 (2026-09-17): a scored chart call older than the music_reconcile_post_max_age_days knob (default 10 days after the chart date) was recorded in the ledger and on called_shot_scored and deliberately NOT composed — a late re-score (the re-opened EP verdicts, a queue drained after an outage) is not news. Emitted so a stale queue reads as “scored, not posted” rather than as composes that never ran |
| music_desk_scored phase=shape_reject | cogs/music_desk.py | guild_id, reason, matchup (the lane), post_preview — a composed line DROPPED by a deterministic guard before the self-gate. reason=self_correction through ship_guard.drop_self_correction (the 17 lanes on the shared guard, every <surface>_scored kind) carries detail.via since 2026-10-18 (B4 of #3410): the phrase lists are the trigger, Haiku ClaudeClient.shows_working decides (claude_api purpose=shows_working), and via=rule means the model gave no answer and has_self_correction decided. Measured: 4 of 11,294 real composed posts in 30 days trigger a call. A rise in via=rule is the model failing; the desk_shape_reject panel splits by via. reason=identity_subset (#3019, every lane) means the take ranked or picked out acts by gender (“SZA is the only woman in the top five”) and the lane’s source does not state that category; constitution.py forbids identity inference, so the guard (output_checks.has_identity_subset_claim) fails closed and detail.phrases names the wording. The source is the lane’s own input where it is open text (the board blob, the milestone and reveal VERIFIED qualifier, the chatter thread, the record card); the structured-feed lanes pass none, because their rows carry no gender field. Measured before shipping: 0 of 6,493 distinct shipped desk lines (30 days) flag, so a non-zero count is a real attempt – read the phrase and the lane’s block. It shows on the existing desk_shape_reject panel (split by reason); no monitor, because the guard already drops the take and the panel shows any rate. reason=self_correction means the response showed the model redrafting in the open (a draft, then “Wait, I can’t say that”, then another draft — the 0.6 gate scored that 0.90 and was right to, since it grades whether claims are GROUNDED, not the SHAPE of the response). market_drop_scored reason=decline_narration is the market DOWNWARD-move guard (ship_guard.drop_market_decline, owner steer 2026-09-18: “Never say dropped don’t narrate negative changes just say is at”): the take narrated a fall in a market number (“first-week forecast has dropped to roughly 61.5K”, shipped to X 2026-09-18 at a 0.9 gate score, grounded in the surface’s own earlier line), detail.phrases names the words; the same reason on market_filtered is the alert’s copy (has_market_decline). The cause was cut first (the drop blob withholds a downward move, the compose no longer allows “the direction as color”, the self-dedup history says the earlier figure is not part of the new line), so a NON-ZERO rate here means the model is still writing the decline off something it is handed – read the phrases. market_drop_scored reason=cross_source_comparison is the same guard on the drop surface – its projected RANK card reports HITS’ rank and units for a Kalshi market’s release, so it is the one other surface holding both publishers at once. music_desk_scored reason=cross_source_comparison is the one guard with NO lane set (owner steer 2026-09-16: “never compare kalshi and hitsdd numbers”): the take weighed a prediction-market first-week figure against HITS Daily Double’s figure for the same release. Both numbers are grounded, one in each block, so the self-gate ships the shape – “priced at ~78.4K first-week units on Kalshi, ~6.4K above HITS’ projection” scored 0.88 and posted on 2026-09-16, and the projection lane shipped ten more over 2026-09-05..07. The compose blocks are the first fix (each board now composes on the ONE source it publishes, and _PROJECTION_FRAMING no longer leads on the divergence), so a NON-ZERO rate here means a lane is still handing a model both publishers’ numbers – read it as a block to find, not only as a guard firing. reason=career_claim means a bare-row lane (chart / radio / called_shot / reconciliation / watch_chart / debut_sales) ranked a release against the artist’s other work, which its block can never support; measured over 40 hours of real posts, 2 of 38 desk posts leaked one from the chart lane and both reached the room. NOT applied to reveal / milestone, whose context stack can genuinely ground a catalogue fact. reason=decline_narration is the chart-POSITION lanes’ second shape rule (owner steer 2026-09-05 “never call it a slip just say where it is”, then 2026-09-06 “never say sliding down, big fall, dont narrate the downwards movement just say where it is”): a take on chart / watch_chart (the desk’s _POSITION_LANES, via _unshippable) or the newsroom’s chart lane (music_news_scored, via ship_guard.drop_decline_narration) narrated the DOWNWARD move (“sliding down from its #8 debut”, “drops four spots to #16”, “went from #8 to #16”) instead of stating where the title sits; the matched wording rides detail.phrases. It exists because the judge grades whether a claim is TRUE and a narrated decline usually is: the 2026-09-06 leak scored high for its “grounded context (debut peak included)” and shipped to X. Not on chart_exit, whose story is the departure. Measured on the real inputs with the real model, n=6 per arm: the old newsroom framing leaked 5/6, the cause reword 0/6 with or without a banned-word list (the list was cut as scar tissue), and the final framing plus the chart lane-hint reword 0/8, so expect a rate near zero; a sustained rate on one lane means that lane’s framing lost to its own context (the newsroom hands the model the debut and peak in its BACKGROUND block, which is where the path narrative comes from). The desk_shape_reject panel groups by reason, so the new reason appears with no dashboard change; no monitor, for the reason given below – a guard firing is the guard working. reason=prior_position is the FALL take’s third shape rule (owner steer 2026-09-18: “Don’t narrate previous position if it was higher. Just say current position.”): a take on chart / watch_chart whose row FELL (the desk passes fall_rank, the take’s own rank, for a _FALL_KINDS / top_fall row only) or the newsroom’s chart lane on a settled weekly row whose last_week is the smaller number, or, with no settled row, on a post whose own single move marker states a fall (“#3(-1)”, “falls 3-5”; mn.wire_fall_position, #3859) (music_news_scored, via ship_guard.drop_prior_position) named the previous position, the earlier week, where the title sat before, or any chart position other than its own – the PEAK included; the matched wording rides detail.phrases. It exists because the decline guard matches a decline VERB and “Man I Need by Olivia Dean sits at No. 10 on this week’s Billboard Global 200, after ranking No. 9 last week” (2026-09-18 08:10Z) has none; the judge scored it 0.85 and praised the clause as “correct per ground truth”. Its own value, not decline_narration, so the two leaks stay separable on the panel: a decline verb and a bare previous position have different framing fixes. The cause was cut first (the two desk fall framings no longer hand over the previous rank or the peak, measured 0/6 and 0/5 on the real row after), so expect a rate near zero; a sustained rate on one lane means that lane’s framing lost to its own blob, which still prints last week’s rank and the peak because the self-gate grounds on them. Same panel, no monitor, same reason as above. The board-shaped lanes (standings / chart_board / chart_proj) take only the ALL-TIME half of that check (has_alltime_claim): a ranked board hands the model every position an act holds and names the highest, so “her highest entry sits at #13” is read off the block, while “her highest-charting album yet” is still invented. reason=attribution_tag is the publication-card lanes’ credit guard: the take names its source in the banned press-release form (“, per HITS” — has_attribution_tag) and survived the one name-the-phrase re-ask (_detag_credit); the dry run measured the raw leak at 5/5 HITS-credited takes and the re-ask cleared all 5, so a nonzero steady rate here means the re-ask stopped working. reason=row_tally grounds rather than bans on the music desk’s board lanes (#2662): the guard rejects a count of a board’s rows because the block hands over each row alone and never a total, but the STANDING board states its top-10 count in the block and leads with it. Measured on four real composes, all four matched the flat ban and would have died before the judge, so the board would have shipped nothing. _drop_before_gate now takes the block and ungrounded_row_tally passes a tally the source states; a count it does not state still drops, and every lane that passes no source is unchanged. A rising rate here on watch_board means the compose started naming a top-10 count the board never handed it. reason=row_arithmetic is the board lanes’ own guard (#2031) — a take that SUMS the board’s rows (“more than the next two combined”). The board gives each act its own number and never a total, so a combined claim is arithmetic the model did itself, and the live dry run caught it doing one that was false (11 spots “more than Drake and Ella Langley combined”, off rows reading 7 and 5) while the 0.6 judge scored it 0.92 and printed the arithmetic in its own reason. Same shape as career_claim: every INDIVIDUAL number is grounded, so only a deterministic rule catches the derived claim. reason=self_correction now rides THIRTEEN more surfaces (#2755 + its class sweep). The checker was always shared, but only three cogs CALLED it – music_desk, cinema_desk and sports_boards, each through its own private _unshippable. Every other surface that composes model prose and cross-posts it to X ran no shape check at all, so a self-correction on any of them shipped. utils/ship_guard.py is the one wiring home, and it deliberately emits on the CALLER’s own <surface>_scored kind with this same phase=shape_reject reason=self_correction shape, so market_drop_scored, music_news_scored, pop_desk_scored, music_alert_scored, sports_desk_scored, cinema_news_scored, bet_alert_scored, bet_board_scored, bet_value_scored, discourse_scored and music_scored join the desk_shape_reject panel and the ops monitor’s NON_SCORE_PHASES rule with no dashboard change and no new field against the budget – every field it writes is already in SHARED_FIELDS. matchup carries the lane, or lane: subject where the surface tracks one. TWO cogs are deliberate exceptions and do NOT appear on that panel, because neither has a <desk>_scored event for the helper to emit on. x_mentions counts its declines by reason on the single x_mentions event, so it calls the checker directly and emits reason=self_correction there beside row_tally and declined. market_alert has no *_scored event at all, so each of its three compose lanes records the drop on the event that lane already uses: market_filtered reason=self_correction on the main alert lane (the same pre-ship drop event its #2547 number backstop uses), rt_reconcile on the reconcile lane, market_outcome on the outcome lane. Watch all three – a gap in one lane is the whole bug. discourse runs FOUR compose lanes and the ICEBREAKER is the fourth – its drop rides discourse_icebreaker reason=self_correction, not the panel, because that is where that lane’s other drops already are. discourse_scored DOES ride the panel, but note its drop does not silence the slot: a self-corrected line takes the same fallback path an empty compose does (discourse_fallback reason=self_correction, then the quip or the icebreaker), so the shape reject and the fallback are two events about one moment. Expect a rate near zero: over the full 30-day population of 10352 real composes on the helper-wired surfaces the guard fires 5 times. Four are market_drop and were already stopped downstream. The fifth SHIPPED, and it is the reason the class sweep mattered: on 2026-08-16 a discourse take carrying “Wait, let me pull the real link.” between two drafts scored 0.78 against the 0.6 floor and crosspost surface=discourse fired four seconds later. The one surface with a measured live leak was the one #2755 did not list. Read a sustained rate the same way as on the desk – a lane’s framing losing to its own context. This rate was NOT observable before the change, so the first weeks of it are a baseline being established, not a regression. The one shape to watch is a take that OPENS with a marker word. The line-leading arm anchors on ^ as well as a newline, so a take whose FIRST word is wait / hmm / correction / scratch that / hold on / on second thought / actually, no is read as a self-correction. On a chart desk that never happens; on the CONVERSATIONAL surfaces this change newly covers (discourse, music) a rhetorical “wait, …” opener is much closer to the persona’s real voice. Measured before shipping: an adversarial set of 20 hand-written legitimate lines flagged 8, but zero of the 10352 real composes in the 30-day window open with a marker word, so the risk is real in principle and has no observed incidence. If a compose-prompt change ever makes her open that way, this guard starts eating good posts and the tell is a shape_reject rate that climbs on discourse_scored / music_scored while the desks stay flat. No monitor, for the reason stated below: a shape_reject firing is the guard WORKING. reason=metric_word:<word> (#2161) is the same shape on the ALBUM-METRIC axis, and it is the one shape_reject reason music_alert_scored also carries (phase=shape_reject, the alert cog’s first use of this marker). One album carries a Pure Sales ladder AND an Album-Equivalent Units ladder at the same time, and the two figures differ by an order of magnitude — live on 2026-08-08 the Karol G album read ~3.1K pure sales and ~45.8K equivalent units in the same tracking week. A take that quotes its number exactly as given but names the OTHER metric (“3.1K units” off the Pure Sales ladder) reads as fully grounded, so the 0.6 gate ships it — it scored that exact take 0.72. output_checks.metric_foreign_word (since CB-12, #4557, through utils.album_metric.foreign_metric_word: the word table finds the word and Haiku claude_api purpose=album_metric reads the line’s metric; only a read of its own metric clears it, and no answer keeps the drop; panel album_metric_read) keys off the READING’s metric rather than the lane, which is why it sits beside _unshippable instead of inside it (_drop_before_gate only ever sees the story name). Measured against the 34 real shipped album-market posts in music_alert_state: it flagged the 2 wrong ones and left 32 alone. The desk_shape_reject panel (added in the same change) is where every reason on this phase is graphed, split by surface — until then nothing graphed shape_reject at all: desk_quality filters it out as a non-verdict and desk_declined counts it in neither column, so a take the model DID write and a guard refused was the one compose outcome with no dashboard home. No MONITOR yet, deliberately: a shape_reject firing is the guard WORKING, and there is no calibrated rate that separates “the guards are earning their keep” from “a prompt regressed” — the panel builds that baseline first. Expect a rate near zero — the live rate was 2 of 15 pure-sales alerts, so a SUSTAINED nonzero rate means a compose prompt started conflating the two metrics again, and a jump to most pure-sales posts means the metric label stopped resolving (check luminate.event_metric). A rising rate means a lane’s framing is losing to its own context. reason=ungrounded_number (#2081) is the same shape on the other figure surfaces – music_news_scored, cinema_desk_scored, music_desk_scored (every number lane, #2543/#2559), music_alert_scored (both compose paths, #2547), market_drop_scored (#2547), sports_board_scored, sports_desk_scored, pop_desk_scored, bet_alert_scored, and (as market_filtered reason="ungrounded_number", its existing pre-ship drop event) market_alert (#2547) carry it, under the lane’s own phase/kind rather than shape_reject, with the flagged values in detail.numbers. On the music_desk REVEAL/tracking lane there is a SALVAGE step before the drop (_reask_ungrounded, owner call 2026-08-25): a production check found good debut cards (ENHYPEN, Stella Lefty) suppressed slot after slot because the model kept adding an invented comparison (“ahead of the 110K the chart accounts project”), so the lane now re-asks ONCE – naming the exact figure and asking for the same read without it – and ships the retry only when it comes back clean AND still carries a MARKET-SCALE figure the draft had (a grounded value >= 1000, so the retry cannot drop the headline units/streams number and ship a numberless take beside a number card; #2571 review) (verified salvaging 5/5 on a real-model dry run). A successful salvage emits its OWN event – music_desk_scored reason="ungrounded_salvaged", shipped=True, the flagged figures in detail.numbers (#2571 review round 4) – so the compose-fabrication rate the salvage fixes stays visible to the ops monitor instead of being masked. A music_desk_scored reason="ungrounded_number" on the reveal lane therefore means the re-ask ALSO failed, so a nonzero rate there is now a stronger signal; a rising ungrounded_salvaged rate means the compose prompt is inventing figures more often even though the card still ships. The market-family surfaces (market_drop / market_alert / betting_alert) and the desk surfaces ground through the shared output_checks.numeric_grounding — the source plus a bare copy of every digit-run it holds — so a compact token (“24h” paraphrased as “24 hours”, a milestone “50th” written as “50”) is not a false-positive drop; market_drop/market_alert also add the market title/topic. market_alert + betting_alert deliberately WITHHOLD the prior price from the blob (#1140), so the backstop there also deterministically enforces “no movement number the model was not handed” — a take narrating “cratered from 50%” drops. A real-model dry run over all five surfaces flagged 0 false positives. It fires when the composed line states a number the ground-truth block does not (output_checks.ungrounded_numbers). It exists because a fabricated figure that SHIPS is the one failure this ledger cannot otherwise report: it shipped because the judge scored it HIGH, so there is no low score to flag and no error to triage, and no after-the-fact check is possible either – these events carry post_preview but deliberately never the ground truth (data minimization), so the comparison exists only at compose time. Measured against the real #2069 and #2076 fabrications it caught 4 of 4, and against the real posts that shipped in both dry runs it flagged 0 of 13. A NON-ZERO rate is the interesting signal here: it means a compose invented a figure and the judge would have been the only thing standing in front of it. reason=career_ordinal (#2154) is its words-not-digits twin, on music_news_scored only, with the flagged counts in detail.career_ordinals. ungrounded_numbers reads DIGITS by design, so a career tally written as an ordinal was invisible to it – and the #2152 re-roll actively SELECTED for that phrasing by rejecting every digit-carrying take, which is how “Cardi’s fourth Diamond single and Megan’s first” shipped to X at a judge score of 0.92. The check is GROUNDING, not phrasing (output_checks.ungrounded_career_ordinals): a wire block that states “her sixth consecutive #1” grounds the take’s “sixth”. That distinction is load-bearing – banning the possessive-plus-ordinal SHAPE was measured against 900 real shipped takes and would newly block 23, all but the WAP post true and sourced, repeating the mistake has_alltime_claim had to undo in #2031. Shipped as a MEASUREMENT first and promoted to a gate on its own rule: it flagged 1 shipped take in 74 over 30h (~1.4%), the low-rate outcome that makes the gate cheap. Like the digit guard it marks the post SEEN, so the story is not re-rolled until a take omits the count. Expect a rate near zero; a sustained one means a compose lane started inventing career tallies. phase=chart_run is the per-release chart-run milestone that replaced a long-runners board (#2031): measured on the real Billboard 200, that board’s top ten was 10/10 identical week to week, so it was cut in favour of a rung that fires once per release. It rides the shared songstats_milestone_state store under a bbrun: entity prefix with metric=chart_weeks, so no new table; a rate of ~3/week across the Hot 100 + Billboard 200 is normal, and 0 for weeks on end means the seed pass never completed. phase=chart_board / phase=debut_sales / phase=chart_proj are the PUBLICATION card lanes (owner steer 2026-08-06, utils/chart_boards.py): the weekly top-10 + debut-sales cards, the individual first-week-units cards, and the four daily projection boards. They ride this same event with no new fields and join the per-lane lane_dark health automatically (music_desk is in LANE_PHASE_SURFACES). Expected cadence: chart_board ~4/week (three top-10s + one sales board), chart_proj 4/day (the HITS variants, each once daily, rotating least-recently-posted first; the owner’s “weekly and daily ofc” steer) — a silent chart_proj day means the HITS read is dark (check hits_fetch) or the document has no chart date. phase=lifetime_board is the LIFETIME record-boards lane (owner ask 2026-08-19, utils/lifetime_boards.py): the evergreen career-tally leaderboards (“most Rhythmic Airplay No. 1s”). It rides music_desk_scored/music_desk_posted with NO new event or field — the desk ship-rate and lane_dark panels group by phase and pick it up automatically. Its data is CURATED (a cited registry, guarded by lifetime_boards.validate), so there is no fetch to fail and no fail-open integration to watch — no dedicated MONITOR is warranted; the shared desk ship-rate covers a compose regression. Expect it QUIET: at most one board a slot, and a board posts ONCE on first sight, then never again until its RANKED ROWS change. The lane keeps a DURABLE per-(guild, experiment-stage) signature — a hash of the ranked artist:value rows (lifetime_boards.signature, which deliberately EXCLUDES as_of, so a date-only refresh does NOT re-post) — in kv_cache, and skips a board whose stored signature still matches. A real change (a new figure, a new leader, a new act) is a new signature and posts again. The state is written from the successful-delivery / intentional-dedup path, not at compose, and the STAGING stamp is separate from the PRODUCTION one, so a #bot-logs audition never suppresses the room after promotion. It is on _BOARD_LANES for the row-sum/row-tally guards but EXEMPT from has_alltime_claim (_ALLTIME_EXEMPT_LANES), because stating the all-time record IS the surface — the same carve record_card gets, but this board keeps the row checks. |
| cinema_desk_scored phase=source_thin reason=opened_film | utils/cinema_numbers.py (fetch_cinema_stories) | guild_id, shipped=false, score=0.0, reason, matchup (the skipped films, 80 chars) — the tracking lane skipped a pull that names a film whose opening result is already out (#4475). No compose ran, so source_thin keeps it out of quality stats; source_dark counts it against the guild’s cinema_desk_posted rows, and the desk posts other lanes, so it does not fire. Expect it on most Friday-to-Sunday slots, and rarely mid-week. A mid-week run of it means the dated query is answering for the wrong weekend again. |
| cinema_desk_scored phase=source_thin reason=critics_posted | cogs/cinema_desk.py (_pick_stories) | guild_id, shipped=false, score=0.0, reason, matchup (the film, 80 chars) — the scores story lane dropped a film whose critic scores already posted, on the critics board or as an earlier scores story (#2517). Both lanes read and write the one critics_board milestone stamp. No compose ran, so source_thin keeps it out of quality stats. Expect it rarely: only when the week’s lead new release is on both the weekend chart and the TMDB now-playing list. |
| <surface>_scored reason starting IDENTITY: | claude_client.py (_IDENTITY_RULE) | no new field. music_desk_score scored the post 0.0 because it sorts acts by gender in a way its ground does not state (#4261). Every lane that calls music_desk_score emits it: the desk’s music_desk_scored, the newsroom, the alerts, releases and X mentions on their own *_scored events. Since #4276 drop_score carries it too (_with_drop_fences, with a pass line for sports event names), so market_drop_scored, music_alert_scored, bet_board_scored, bet_alert_scored, sports_board_scored and the market_alert / market_outcome rows can carry it. Count them with ['tootsies'] | where event endswith '_scored' and reason startswith 'IDENTITY:' | summarize count() by event, bin(_time, 1d). |
| <surface>_scored reason starting WORKING: / HEDGE: | claude_client.py (_SHOWS_ITS_WORKING_RULE, the music-desk CONFIDENCE DISCLAIMER line) | no new field. The self-gate judge (music_desk_score, drop_score, cinema_desk_score, and winner_line_score since #4228, on the market_outcome row) scored the post 0.0 because the post shows the model’s working (a refusal, a search, a self-correction, talk about its task) or doubts its own figure (#3427). A forecast stated as a forecast is not a HEDGE: (#4289); a HEDGE: row on phase=='tracking' whose post has no doubt word is the regression to watch for. Count them with ['tootsies'] | where event endswith '_scored' and (reason startswith 'WORKING:' or reason startswith 'HEDGE:') | summarize count() by event, bin(_time, 1d). The wheel’s form of the same judge is a claude_api row with purpose="decline_narration". |
| music_desk_scored phase=riaa_plaques reason=plaques_recent | cogs/music_desk.py (2026-10-23, #4683) | guild_id, shipped=false, score=0, matchup (the act), detail{chart, last} — a per-act NEW PLAQUES board skipped BEFORE compose because the same act and format (detail.chart) already posted a board in the last 7 days (_RIAA_PLAQUES_ACT_GAP_SECONDS; detail.last is that board’s UTC date). The board window is 3 days, so one new award used to redraw every award still in it: Zach Bryan and Wiz Khalifa each posted their singles and album boards on two days out of three (2026-10-21 to 10-23). The mark is <guild>:<stage>:riaa_plaques:recent:<chart> in lifetime_board_state, written on a confirmed delivery with a 7-day TTL. A steady rate on a busy cert week is the guard working; a board that never posts again for an act means the TTL is not expiring. |
| music_desk_scored phase=riaa_plaques reason=plaques_twin | cogs/music_desk.py (2026-09-05) | guild_id, shipped=false, score=0, matchup (the skipped act), detail{chart, twin} — a per-act NEW PLAQUES board skipped BEFORE compose because its signature already shipped under ANOTHER credited act of the same collaboration batch. awards_by_act hands every act on a shared credit the same rows, so “Intro” 7x + “Ganga” 4x boarded as Peso Pluma at 23:04Z on 2026-09-04 and again as Tito Double P at 00:02Z, one signature under two act keys; only the text dedup stopped the twin. The board now keeps a second fire-on-change state keyed on the signature alone (<guild>:<stage>:riaa_plaques:sig:<sig> in lifetime_board_state, value = the chart that shipped it, written on the same confirmed delivery or dedup close-out as the act’s own key). detail.twin names that chart. Expect it rare (a collaboration-only batch between two recognised acts); a steady rate is the guard working. The SAME night also showed the other way a plaques board repeats: #2957 renamed its chart key (riaa_plaques:<act> -> :single), so every open batch read as never posted and Nickelback’s board went to the room twice (15:07Z and 02:08Z, the same signature under both keys). No event marks that class; the tell is a music_desk_posted phase=riaa_plaques for one act twice inside the 3-day window with no new award between them. |
| music_desk_scored phase=catalog_stat / phase=daily_hits reason=rank_names_act / rank_wrong_place / rank_wrong_metric / rank_unchecked | cogs/music_desk.py (#3517, 2026-09-20; CB-11 #4557, 2026-10-19) | guild_id, shipped=false, score=0, matchup (the act), post_preview — a composed CATALOG-COUNT or DAILY-HITS caption dropped by the lane’s own rank fence (music_desk._catalog_caption_reason: the exact checks in catalog_stats.rank_claim_mismatch, then Haiku catalog_caption_check, purpose catalog_caption, for the open shapes; rank_unchecked = Haiku gave no answer, so the caption drops, fail closed; handed to _milestone_candidate as caption_check and run right after the shape guard, BEFORE the number backstop, the ranking guard and the paid self-gate, so a rejected line is recorded once, here, and never also as a shipped score). rank_names_act: the caption names an act the field holds above or level with the subject (exact match), OR Haiku reads a named comparison with any name (“trailing only X”, “right behind X”, “second only to X”, “only X has more”; the phrase regex that read these until CB-11 missed “right behind” and falsely hit “after only six years”) — the shape the owner cut on 2026-09-20; the material states the place only, so the comparison is invented, and checking the name alone would pass an act the model made up. rank_wrong_place: the caption states a COUNT place the lane did not prove (“3rd most among …” off a 4th-place rank, “the most among …” off anything but first); the place syntax is scoped to the count ranking (“Nth most among / of any”, “#N among”), so the standing block’s “#6 most-streamed artist” and the top song’s “most-streamed” are not read as places. rank_wrong_metric: Haiku reads a place put on another metric (“Spotify’s fourth-most-streamed artist”, “fourth in monthly listeners”). The shared ranking guard (ungrounded_ranking_claims) keys on a ranking CONTEXT and passes all of these shapes (Codex review on #3517, measured), which is why the lane reads the place itself. The fence runs ONLY on the deterministic-rank path: a ROUND count composed off a reported web qualifier states that source’s place, not the field’s, and gets no check. The rung is recorded on delivery, so a dropped act re-composes next slot. Expect it rare (the live dry run on #3517 gave 0 of 4); a sustained rate means the compose is reaching past its material — read post_preview. No panel or monitor of its own: it rides the existing music_desk_scored shipped=false drop panel and the per-reason breakdown there. |
| music_desk_trim | cogs/music_desk.py (#3563, 2026-09-21) | reason, kind (the lane), tier, metric, guild_id, and channel_id on streams_tier; detail{artist, rung} — ONE event per card the NOTABILITY TRIM refused. The owner asked for it with the trim itself (“make sure we have metrics for this — what we’re cutting and how much we’re cutting each”), so the change can be reviewed a week after it ships rather than argued about. This row was rewritten once because successive fixes left it contradicting itself; keep it coherent rather than appending. The reasons. unwatched_move is a plain chart MOVE by an act the watch does not carry, from the 2026-09-21 volume review of a single day’s posts; measured live it refuses about 16 a sweep against 38 stories kept, naming the acts the owner had deleted by hand. That review’s other cut, the surge band from #50 to #20, is deliberately uncounted — a counter for it measured 116 refusals a sweep, because without the band’s previous value it counts every big mover anywhere on the chart, which reads as cards cut and is not. Read its volume from the before-and-after watch_chart posting rate, exactly as with chart_exit. streams_tier is the artist-tier gate on the STREAMS family only — a first-week, album-equivalent or pure-sales forecast never meets it, by owner scope. light_milestone is a 100M/250M/500M track-streams rung on an act outside the daily top 50. tenure_floor is a chart-tenure rung under 730 days or 100 weeks. release_age (2026-10-20, utils/recency.py) is the release-age cut: a single-title chart card or chart milestone outside the top 5, or a 100M/250M/500M streams rung, whose title is more than two years old. Its tier names the half that refused it: deep (below No. 10 and not a debut, cut on position with no lookup), old (dated, older than MUSIC_DESK_RECENT_MAX_AGE_DAYS), no_date (the resolver had nothing). kind is the lane: watch_chart and chart count OFFERS (the cut stamps nothing, so a held position is offered every slot; read dcount of the act per day), and the milestone stories count CARDS for deep / old because the rung settles. Since the same day the ALERT cog’s daily lanes (cogs/music_alert.py _recency_cut) count here with kind = the lane (apple_jump, radio_riser, itunes_debut, …): the climb lanes take a top-20 line (MUSIC_RECENT_CLIMB_MAX_POS), every other lane the top 10, and a cut marks the title out of tries for the day (_refuse_today), so each cut is offered once a day. The newsroom’s chart stories count on music_news_filtered reason=release_age instead; the release_age_cut panel on the desks board reads both. fall (2026-10-23, owner steer “only rise”) is the fall cut: a title that FELL gets no single card. On the watch lane (kind=watch_chart) it is a top_fall inside the top ten; on the Billboard lane (kind=chart) it is a fall / top_fall that passed _move_is_cardworthy, so a deep fall the rank cap already refused is not counted. On both lanes _count_fall_cut applies the release-age rule first: a fall that rule refuses still counts as release_age, so fall holds only falls that would otherwise have posted (Codex on #4679). A chart with a top-10 board (platform_top, chart_board) shows the fall there, and the artist boards still read falls; Global Excl. US, Spotify Global Weekly, US radio and US YouTube have no top-10 board, so their falls are not shown at all (about 10 cards a week before the cut). It counts OFFERS: nothing is stamped, so a held fall is offered again each slot; read dcount of the act per day. Its before-and-after volume is the watch_chart posting rate (108 position-only cards in the 7 days before the cut). itunes_depth (2026-10-23, owner steer “lets cut itunes to top 5”, “for both”) is a single iTunes card below #5 (x_gate.ITUNES_TOP_N, which the X gate reads too): the desk’s watch lane counts it with kind=watch_chart (OFFERS, no stamp), and the alert cog’s iTunes lanes count it with kind = the lane (itunes_jump, itunes_debut, …). Neither stamps anything, so both count OFFERS (about one per tick while a title sits below #5), and a title that climbs into the top 5 later the same day still posts (Codex on #4679). The newsroom’s iTunes stories take the same cut as music_news_filtered reason=itunes_depth. The iTunes top-10 boards still post. Measured on X before the cut: 28 of 77 iTunes posts in 2 days sat at #6-#10. What counts as a card cut, and what does not. tenure_floor settles its rung on the way out (_settle_rung_quietly), so it fires once per crossing and its raw count IS cards. light_milestone settles only when tier=off_top_tier, a DECIDED cut: the gate runs only once ranked_to proves the watch reached the tier, so an act placed inside the daily prefix past the line, or absent from a WHOLE page, is genuinely not in that tier. Its other three tiers are UNDECIDED, do not settle, and therefore repeat per slot and per channel: listeners_only (the act was placed only among names the monthly-listeners page contributed, pos > daily_count, so that rank says nothing about daily standing) and short_page (the act was absent from a daily page that reached the TIER but not the watch size, so absence is not evidence — an act at real rank 61-100 is simply missing from a 60-row page), and total_top (the act’s kworb TOTAL-streams rank is inside the tier, so a daily position past it may be a “Daily” under-read, #3868). streams_tier never settles anything: it fires inside _pick_readings, which re-reads the live Kalshi set every slot, and the compose path runs once per configured channel plus once more on a 429 retry. So: group by tier, and read dcount of the act per day rather than a raw count for streams_tier and for the three undecided milestone tiers. The desk_trim panel groups by reason, tier, kind per day for exactly this reason. Adding the tiers together reads a watch-resolution failure as noise removed, which is backwards. What the streams_tier count excludes, each for a measured reason: the pick loop stops at its per-slot limit, so counting inside it would measure where a refusal ranked rather than how much the gate holds back; the raw sweep carries stale reveals and duplicate tickers rank_desk drops, which this lane could not post anyway; and an already-carded event (reveal_seen / exclude) is held by the dedup, not by this gate, so counting it would re-bill the gate every slot of the dedup window. Two lanes are deliberately uncounted. The CHART EXIT lane returns before it reads Billboard, so a counter would mean doing the work the switch turned off; its volume is the 48 music_desk_posted rows from before 2026-09-21 — 12 a week — which ride phase, not kind: where event == 'music_desk_posted' and phase in ('chart_exit','chart_exit_board'). The ALERT cog applies the same tier gate and counts nothing: its tick polls every 10 minutes ahead of every eligibility check and the per-tick post cap, so its refusals are not cards. Be precise about what that leaves uncovered — the desk counts what THE DESK could post, so a streams market rank_desk drops is refused on the alert lane and lands in no count at all. The miss-verdict change has no counter either: it changes what a card says and cuts no card. Baseline, measured on the 28 days BEFORE the change: the desk posted 1531 cards (383 a week); the tier gate cut 11 of 111 live Kalshi readings and 0 of 42 units/sales readings; 23 of 30 tenure cards sat under the floor; 13 of 26 light track-milestone cards named an act outside the top 50. No monitor, deliberately — a gate firing is the gate working, and the failure this trim could cause is the desk going QUIET, which the ops monitor’s existing surface_dark / lane_dark findings already cover. The signal to watch is the opposite of a spike: all three reasons at ZERO for a week means the kworb watch is failing open (an empty watch reads every act as recognized), not that the noise stopped — read chart_fetch before reading a zero as success. |
| music_desk_clause_cut | cogs/music_desk.py (#3257, 2026-10-17) | post_preview (the line BEFORE the cut, first 120 chars); detail{clause} — ONE event per take whose trailing clause _cut_flourish removed: output_checks.trailing_clause found a digit-free last clause, and Haiku (claude_api purpose=flourish_check) answered FLOURISH. The post still ships without the clause. No guild_id: _detag_credit does not carry one, so a count over two destinations counts each take once per destination. Expected under 1 a day (18 of 5,939 shipped lines in 30 days, from about 37 flourish_check calls a day). Panel desk_recompose_cut. No monitor: a cut is the check working, and a check that stops working fails toward today’s behavior (the clause ships). |
| music_desk_scored reason=dedup_memo (score 0.0) | cogs/music_desk.py (#3581, 2026-10-17) | guild_id, phase (the board lane), matchup (<kind>: <subject>), shipped=false — a MARKER, not a judge verdict: _compose_board_unit skipped a board whose same headline the post-dedup gate dropped within _DEDUP_MEMO_HOURS (4h). No line, no judge; ops_monitor.NON_SCORE_REASONS keeps it out of quality stats. Read it against post_dedup on the same phase (panel desk_recompose_cut): the platform lane’s drops fall from about 50 a day to a few, and these skips take their place. |
| music_desk_scored reason=census_told (score 0.0) | cogs/music_desk.py (#3155, 2026-10-17) | guild_id, phase (standings or artist_catalog), matchup (<kind>: <subject>), shipped=false — a MARKER, not a judge verdict: _census_told skipped a census board or an artist board whose fact key (chart_standings.census_fact_key: the act, the chart, the printed chart week and the act’s set of titles) is already in the SHARED wire_chart_census store. The desk’s census and artist boards, the X-mentions boards lane (room posts) and the newsroom’s whole-chart count relay (production) write that store. No line, no judge; ops_monitor.NON_SCORE_REASONS keeps it out of quality stats. Query: where event == 'music_desk_scored' and reason == 'census_told'. |
| music_desk_scored phase=pollstar_board / phase=radio_format_board reason=stale_chart | cogs/music_desk.py (2026-09-06) | guild_id, shipped=false, score=0, matchup (N of M boards, dated <oldest chart date>), detail{charts (the stale chart keys), age_days} — ONE row per Pollstar lane per slot when a board’s chart is older than pollstar_boards.MAX_CHART_AGE_DAYS (10); the boards yield BEFORE compose. This is the freshness signal for a feed that stands still: pollstar_fetch reads ok=true because the data arrives (it is just old), so without this row a stuck upstream looked like a lane that was merely deduped. Measured 2026-09-06: every chart at Aug 14/17 for three weeks, and “this week” boards posted off the Aug 17 chart on Aug 30-31. A stale_chart count that persists across days USED to be read as “the upstream, not us; the lane resumes on its own when the feed moves”. That was wrong, and it cost 15 days of two beats (#3357). The lane could not resume: Pollstar publishes every weekly issue under its OWN chart id and never updates one, so the pinned registry ids kept serving 8/14 and 8/17 forever at HTTP 200. The reader now resolves the current issue off each chart’s config endpoint, so a sustained stale_chart count means the upstream genuinely has not printed. It is also no longer silent: the rows are counted by the ops monitor’s lane_stale_upstream (high) finding, and they are excluded from judge-quality stats (NON_SCORE_REASONS) — left in, 304 markers a week pulled the music desk’s measured mean from 0.83 to 0.74 |
| music_alert_scored / music_alert_posted phase=artist_market / phase=artist_market_move / phase=artist_market_leader / phase=artist_market_settle | cogs/music_alert.py | the alert’s ARTIST-MARKET lanes (#2800 line E, #2833), riding the alert’s own scored/posted events with no new field. artist_market is a NEW Kalshi event market about a watched act (listed inside 10 days, 50+ contracts, the leader priced 5%-95% by the shared kalshi_price rule); artist_market_move is the biggest week move on such a market (15+ points, 300+ contracts, off Kalshi’s public daily candlesticks read in ONE batch call per event – market_fetch query=candles_batch:<n>). Both ride the music_artist_markets experiment (STAGING by default: delivered=staged in #bot-logs) and dedup on Kalshi tickers in post_dedup_history (music_alert_artist_market, the event ticker, 1 year; music_alert_artist_market_move, <market>:<up|down>, 30 days; music_alert_artist_market_leader, <event>:<hero label>, 30 days), never the song reuse block. artist_market_leader is the STANDING card, the companion to a move: a shipped move post writes its board to the QUEUE (music_alert_artist_market_leader_queue, <event>:<moved market>, 5 days) and the standing lane – registered BEFORE the move lane, so it reads only an earlier TICK’s queue and the card lands on the NEXT SLOT, 10 minutes on – reports what that board says now. It withholds on a board whose leader is the leg the move card already reported, on a board that has left the open catalog or that the listing lane posted in the last 3 days, on a board whose raw read came back a full page (truncated, so unrankable), on a standing this board already reported last, and on any leader that is not BID-SUPPORTED. A cumulative DATE board (kalshi_ladder.is_date_ladder) states the soonest window, the first even-money window and the widest one instead of a bare ‘leader’. Expect each QUIET: at most one card a day per lane per guild, the standing lane only ever follows a board the move lane already posted, and the move lane fires only when a market about a watched act actually moves (measured 2026-09-02: 3 of 168 open act markets moved 20+ points in a week). A music_alert_scored row with phase=artist_market_move and a preview naming a price the blob did not state is the ungrounded_number guard’s job; a day with neither phase is normal. music_alert is NOT in ops_monitor.LANE_PHASE_SURFACES, so neither lane joins the per-lane dark rule; the silent-failure tell is market_fetch ok=false on query=candles_batch:* or raw_markets:*, which the integration-health rates already count. | artist_market_settle (#2800 line E): the follow-through on a market the listing lane announced – its every market finalized/settled inside 7 days, the YES outcome (or a single market’s NO) as a big-number card; the roster is the listing surface’s event tickers (held a year), each reported once on music_alert_artist_market_settle (permanent), at most 12 event reads a tick; a resolved market whose title names no watched act is skipped. Expect it rarely (a listing resolves months later), so a quiet lane is normal while artist_market posts are young.
| market_news_read | utils/market_outcome.py, cogs/music_alert.py, cogs/market_alert.py | guild_id, surface, phase, ok, reason (found / no_web_client / no_question / web_failed / web_no_result / web_no_publisher / web_unusable / no_event_date / outside_window / moved_leg / news_take_dropped / news_off_subject / news_check_failed), detail.ticker, detail.topic. ONE Perplexity read (pplx_market_move_news, recency week) per market card about to compose, on every lane that reads the news: surface=music_alert phase=artist_market_move (after the daily gate) and phase=artist_market_leader (the standing card, #3760), and surface=market_alert phase=<alert kind> (swing, lead_change, ending_soon, conviction breaks; not the forecast lanes or settled_out). A found, press-cited report LEADS the blob and the take runs the market_news angle; on a date board the standing card moves to the window that CONTAINS the news date. no_event_date / outside_window are the #3760 date-window check. Added 2026-10-02 after the Drake remix card shipped “at 94%, up 65 points in five days” a day after Drake announced the remix (#3783), widened the same day to the standing and market_alert lanes (#3786). Fail-open: ok=false ships the card as before. news_take_dropped is the FALLBACK: a news-led take was refused before delivery (a guard or the self-gate), and the card recomposed once from the price-only blob, so the news never costs a post. A high news_take_dropped share on one lane means the gate and the market_news angle disagree there. A found rate near 0 while pplx_market_move_news stays ok means the parser is rejecting real reads; read the raw Sonar text. No monitor: a miss degrades one card to the old shape, it never posts a wrong one. news_off_subject / news_check_failed (market_alert only, #4682): the Haiku judge market_news_on_subject (claude_api purpose=market_news_subject) reads every found line before the qualifier check. It drops a line about another song, artist, team or chart (the shipped “John Mayer announced ‘Tom and Jerry’ … while Boston overtakes” Hot 100 card). It fails CLOSED: no verdict is news_check_failed, and the card ships price-only. A run of news_check_failed means the Haiku call is failing. |
| music_desk_scored / music_alert_scored reason=ungrounded_age (score 0.0, shipped=false, any lane) | cogs/music_desk.py + cogs/music_alert.py _drop_ungrounded (2026-09-06, owner ask “cut those last sentences on age of the record”) | guild_id, phase (the lane), matchup, post_preview, + folded detail.claim (the phrase). The age-in-WORDS twin of reason=ungrounded_number (#2081): output_checks.ungrounded_age_claim flags a take that states how old the release is (“thirteen years deep”, “a track over a decade old”, “sixteen years after its release”) when NONE of its sources state an age. The digit check already dropped “14 years old”, so the model wrote the age in words; 20 watch-chart takes and 2 Spotify jump posts shipped that way in the 30 days to 2026-09-06, every one off a kworb row that carries no release date. PERMISSIVE by design: a source that states any age in any form (the Luminate head’s catalog, out 4.2 years, a dated debut blob’s came out 18 days ago, an Anniversary edition title, the oldest-songs board) stands the check down – it checks that the source spoke to age, not that the two ages agree. Measured over 30 days of shipped takes (n=5774): the claim pattern matches 67, of which the tracking + lookup_board lanes (18) are grounded and stand down; the rest (watch_chart 18, chart_run 15, milestone 12, chart 2) now drop. Root cause fixed in the same change: the shared _CONTEXT AGE / STAGE paragraph asserted “the head names the release’s milestone” as a fact about every post, and the chart lanes have no age in their blob, so the model supplied one (reworded to a condition; real-model n=6 dry run 3/6 -> 2/6, so the guard carries the rest). In NON_SCORE_REASONS with its twin, so it never reads as a judge score. Expect a low steady rate; a SUSTAINED rate on one lane means that lane’s blob should state the age (or the prompt regressed). |
| music_alert_scored phase=settle reason=settle_retry / settle_retry_cap / settle_rejected (score 0.0, shipped=false) | cogs/music_alert.py _settle_retry_left / _settle_consumed (#3822, 2026-10-04) | guild_id, matchup, + attempt on settle_retry. The Kalshi readings loop’s SETTLE outcome markers. settle_retry = _fire failed for a reason a fresh attempt can pass (a compose or score exception, a Discord delivery of 0, or a deterministic shape guard), so the baseline is kept and the next tick composes again; attempt counts to _SETTLE_MAX_RETRIES (3). settle_retry_cap = every retry failed, so the lane advanced the baseline and gave the settle up. settle_rejected = a verdict (a score under 0.6, a topic repeat, an empty take) consumed the settle; its judge or topic_dedup row is emitted separately. A dropped settle used to leave no row of its own. All three are in ops_monitor.NON_SCORE_REASONS, so they do not count as judge scores. Ticker (#3889, 2026-10-06): every settle row (the phase=settle score row, these three markers, and music_alert_posted phase=settle) carries the Kalshi event ticker in detail.ticker (ticker is a _FOLD_FIELDS name, so no new column). One album has up to three settle markets (streams, units, pure sales) with the same matchup, so the ticker names which market missed. Query: parse_json(detail)['ticker']. Dashboard: music_alert_settles. Query: ['tootsies'] \| where event == 'music_alert_scored' and reason startswith 'settle_' \| summarize count() by reason, bin(_time, 1h). |
| music_alert_scored / music_desk_scored phase=shape_reject reason=release_title (score 0.0, shipped=false) | cogs/music_alert.py _fire (SETTLE only) + cogs/music_desk.py _compose_line (settled reveal only) (#3835, 2026-10-05) | guild_id, matchup (the subject), post_preview. A settled take that names no release title (output_checks.missing_release_title). It reads as the artist’s number, and the 0.6 gate cannot see that because the figure is grounded. On music_alert it is retryable, so a settle_retry marker follows it, and a settle_retry_cap follows if every attempt drops. No new field. The desk_shape_reject panel groups by reason, so it shows with no dashboard change. No monitor: a guard firing is the guard working, and a settle that never ships already shows as settle_retry_cap on the music_alert_settles panel. Expect a low rate: 0 of 200 dry-run settles composed with the real history, and up to about 45% of composes on one album (Popstar units, gpt-cheap) with an empty history; a sustained rate on one album means a title that does not fold to what the takes write (read post_preview against matchup). |
| <surface>_scored phase=declined | utils/events.py (emit_decline), called by the five wire desks (cogs/pop_desk.py, cogs/sports_desk.py, cogs/cinema_news.py, cogs/music_news.py, cogs/cinema_desk.py) and, since #2086, by cogs/music_desk.py, cogs/market_drop.py, cogs/music_alert.py and the three betting surfaces, and since #3163 by cogs/commentator.py (commentary_scored, too_short only) | guild_id, phase (declined), shipped (always False), score (always 0.0), reason (model_declined | empty_after_link_strip | too_short | throwback_composed – on pop_desk, cinema_news, sports_desk and music_news the composed take states a throwback frame (utils.retrospective.date_hook(look_back_only=True)) that the source-post cut did not see, #4697; it carries post_preview + chars, and music_news also marks the story seen; on market_drop_scored also leader_fabrication | leader_check_unavailable, the chart take fabrication gate, #3442: the regex pre-check did not confirm the take names the leader, then the Haiku judge claude_api purpose=market_take_check said False or errored – the gate fails CLOSED, so a judge outage reads as a run of leader_check_unavailable), matchup (the subject, else the lane), and on too_short, throwback_composed, leader_fabrication and leader_check_unavailable also post_preview + chars — the compose model DECLINED to write a take, so nothing shipped (#2051). too_short is the one reason here that is a DEFECT rather than editorial restraint (#3160). It is emitted by the SPINE (cogs/scheduled_poster.py), and since #3163 also by the two autonomous posters outside the spine that ride a *_scored kind: cogs/music_alert.py (both compose sites) and cogs/commentator.py. cogs/market_alert.py has no *_scored kind, so it records the same drop as reason=too_short (with post_preview) on each lane’s own pre-ship event: market_filtered (the alert lane, terminal), rt_reconcile and market_outcome (transient, retried next sweep like an empty compose). Over 30 days to 2026-10-18 the shortest real line on these three surfaces was 51 characters (market drops), 56 (music alerts) and 192 (commentary), so the 15-character floor drops no real post there. The cause the floor answers: the compose stopped mid-word and returned a fragment, and utils.output_checks.too_short_to_ship dropped it before the delivery walk. The floor is 15 visible characters, measured — over 30 days of shipped posts the shortest real scheduled post is 22 characters (“Clairo turns 28 today.”). A post carrying a LINK is exempt at any length, so the three curator surfaces, which ship a bare fixup link as the whole post, are untouched. The cause: on 2026-09-11 compose_pop_take returned the two characters “EM” (a cut-off EMPTY decline). is_empty_response compared against the whole sentinel word and passed it, and pop_desk_score scored the fragment 0.62 — over the 0.6 ship floor — with a stored reason that graded the SOURCE wire headline, because the take gave the judge nothing to grade. The fragment shipped to the pop channel and crossposted to X. The judge is not a reliable guard on this input: re-run live on the same fragment and blob it returned 0.2, so the same input scores either side of the floor. That is why the floor is deterministic. Since these ride the existing *_scored kind, the desk_declined panel and the compose_declined finding cover them with no new event kind and no new field names; compose_declined flags any reason other than model_declined/all_seen, so a run of too_short raises on its own. A decline is a real editorial decision: the model returns EMPTY for a charged or out-of-lane moment, or one with nothing to state. It used to be the ONLY outcome in the whole funnel that left no trace — the self-gate emits *_scored, the repeat check emits post_dedup/topic_dedup, the staleness judge emits news_stale, delivery emits *_posted, and this path just returned None. So “the desk never saw the story” and “the desk saw it and passed” looked identical from telemetry, and they need different fixes; that gap is what made the missed Drake posts (#2033) hard to diagnose. Rides the surface’s own *_scored event rather than a new kind, the same shape music_desk_scored phase=shape_reject already uses, so it costs nothing against the field budget. It is NOT a judge score — score=0.0 here means “there was no take to grade”, so ops_monitor.NON_SCORE_PHASES excludes it and the desk_quality panel filters it out; counting it would report a surface that DECLINED as a surface whose writing is bad, and at a high rate would trip surface_dark and point triage at the judge instead of the compose prompt. surface_dark cannot cover this class at all: it reads judge scores, and a declined take never reaches the judge, so a desk refusing everything shows no scores and reads as a quiet news day — which is why the ops monitor gained its own compose_declined finding (medium, at 60% of at least 6 attempts; deliberately loose, because some declining is correct and only a lurch is a signal). Watch the desk_declined panel: a decline rate that jumps after a prompt change means the compose prompt started refusing real material. #2086 closed the class: the music desk’s nine lanes read is_empty_response(line) or self._unshippable(...), and or short-circuits, so an EMPTY response emitted neither this marker nor shape_reject — silent on the surface that scores more rows than any other. All nine now go through _drop_before_gate, which keeps the two drops as SEPARATE events because they need different fixes: declined means the model refused to write (fix the prompt or the material), shape_reject means it wrote something a deterministic guard forbids (fix the guard or the framing). bet_alert_scored joined ops_monitor.SCORED_KINDS in the same change — it was the one *_scored kind missing while both its siblings were in, so betting_alert also gains the self-gate quality monitoring it lacked. Still NOT emitting: cogs/market_alert.py (it has no *_scored event to ride; its too_short drop is on its own events, above) and the discourse opener path (it carries its own decision events). The reply surfaces (ask, x_mentions, discourse openers, chimein) stay out of the floor: real ask_answered replies run as short as 15 characters (#3163). |
| music_news_chart_lookup phase=first_week | cogs/music_news.py | ok (a projection was settled), phase (first_week), matchup (the subject), + folded detail: charts (hits_projection), chart (the document: midweek-20 / hits-top-50), pos (the projected rank), reason — the newsroom SETTLED a first-week projection claim against the HITS Daily Double document (#2049). The desk already read this document first-party for the called-shot lane, and it is cached, but the newsroom could not reach it — so a first-week projection was the one declared story kind the settler had to relay on account trust, and it is the kind most likely to be wrong, because a projection MOVES through the tracking week. A wire quoting Tuesday’s midweek number on Friday quotes a number the document has since changed. Settles on UNITS rather than a position, so it carries its own reading (ProjectedUnits) rather than a CHART_SOURCES row; latest_projection() is the same call the called-shot lane makes, so both lanes read one number. The extra refusal reason is not_first_week, and it is the WRONG-WEEK guard: an album stays on the document for many weeks, so matching on title and artist alone answers a first-week question with a later week’s number. Caught in the dry run on the real midweek document for 2026-08-15 — Olivia Rodrigo’s album sat at #3 with 75,000 units and last_week=1, while its actual first week read 79,863 on the building chart for 2026-08-08. A sustained not_first_week stream is the guard working. A sustained absent here means the document is not carrying what the wires talk about. STREAM claims settle here too (#2771). #1890 wired kworb into the CHART branch and left the MILESTONE branch reading the open web, where both legs (Perplexity + Grok) inherit aggregators that LAG Spotify – so a TRUE claim came back refuted. A wire said Rihanna had 8 songs over 2 billion Spotify streams; the judge answered “her highest is ‘Stay’ at 1.972B” and the story was dropped three times, while the artist page the bot already fetches ~20x a day listed exactly 8 over the rung, Stay at 2.027B and her highest at 2.648B. Drake’s “140 billion” was dropped four times over four days against a leaderboard reading 140,255,100,000. Rows carry phase=milestone, detail.source=kworb_streams, plus shape (catalog | track | career), rung, claimed, count and total. Since #3415 the row also carries via (classifier | regex): whether the classifier’s stream_shape / rung / count decided the shape or the claim regexes did; the entry-history rows (detail.chart + claimed) carry the same via for entry_count. Read the settled rate by via: a classifier row that reads not_settled far more often than a regex row says the model is shaping claims the page cannot bear out. ok=False is NOT a refutation – it means the page was read and the claim did not bear out, which happens legitimately when a wire count is one ahead on the day a song crosses, so the reading is dropped and the web path decides. Scoped to a claim naming BOTH Spotify and a stream metric, so an Apple Music or YouTube claim is never answered with Spotify numbers. Watch the RATE of ok=True on this phase against music_news_filtered reason=refuted: the fix is working when refuted-stream drops fall and this rises; both flat means the settler is not being reached and the claim shapes have moved. detail.reason=stale (2026-09-30): a track that crossed the claimed rung more than FRESH_CROSSING_DAYS (7) days ago at its daily pace, a track with no daily that is further past than 25M a day could reach, or a career total more than 10% past the rung means the wire re-posted an OLD milestone (@chartmastersorg’s “You Rock My World” at 100M, which kworb read at 366,235,376 and which shipped as “has reached 366.2M”). The row reads ok=False, and the story drops with music_news_stale via=stream_rung_passed. detail.reason=unbounded (#2787): a CATALOG claim read off a page whose every row clears the rung. kworb lists an artist’s songs biggest first and cuts the page off (500 rows at most), so the page may be missing qualifying rows and its count is only a lower bound. The reading settles in neither direction (ok=False) and the web path decides. catalog_stats.count_catalog owns the rule; the settler reads it. Measured 2026-10-17 over every artist this lane and the desk’s catalog qualifier saw in 30 days (64 artists): no page came near it, the closest being Drake at the 100M rung with a smallest row of 15.7M. A sustained unbounded stream means a lower rung went live or kworb shortened its pages. detail.reason=subset (#4583): the classifier’s stream_shape=subset, a total or count scoped to a part of the act’s credits (“as lead artist”, “lead-credit songs only”) or to one period (“in 2026”). kworb is not read, the row carries no rung or total, and the web path decides. Before it, kworb’s all-credit total settled these: 9 fresh @chartmastersorg lead-artist milestones in 30 days were cut as stream_rung_passed (The Weeknd’s “90 billion lead-artist streams” against a 99.5B all-credit total), and 5 more had their claim rewritten to drop “lead”. A sustained subset rate on non-scoped posts means the model is over-calling the shape. detail.reason=undated: the page shows no daily, so the crossing cannot be dated; the story is deferred (music_news_stale via=stream_undated) and not marked seen, the desk’s rule for an undated crossing. music_news_stale via=career_count_passed (#4745): when kworb does not settle a career claim, the verify judge reads Songstats’ career total, and it now CONFIRMS a rung that total sits at or above (it used to refute “8 billion” off 8.6B, 45 of 63 true milestones on the 30-day replay). Songstats reads up to 15% above kworb, so a confirmed count more than 25% past the rung (mn.SONGSTATS_CAREER_STALE_RATIO) is an old milestone: the story drops and is marked seen (folded detail: claim, via, ratio). A confirmed count above the rung never reaches the card or the gate; the post states the wire’s rung. Panel: newsroom_refutes. Web research is withheld from the compose once a stream reading settles, like Songstats. A settled track or career story now states the RUNG (“at or past 100M”), never the live total. reason=hits_older (#3827): a number authority (_post_text_grounds) posted AFTER HITS last edited the document (Projection.edited_before, off _updatedAt), and the post states a count the document does not round to (mn.wire_units_disagree). The post is the newer number, so the reading is dropped and the post keeps its own number. Live case: chartdata’s “#6(new) @Tinashe, Popstar 35,500” (Billboard’s final, 2026-10-04 21:02Z) against a document edited 2026-10-02 22:25Z that still projected 32,386 at #9. A post that agrees (“32K” for 32,386) still settles. |
| music_news_chart_lookup phase=chart | cogs/music_news.py | ok (a position was settled), phase (chart), matchup (the subject), + folded detail: charts (the chart keys read, comma-joined – the classifier’s chart_key when it named a readable chart, else charts_named off the claim’s own words, #3414), via (classifier | regex: which of the two chose the keys – read the ok-rate by via, and a rising absent under classifier on a LIVE chart is the failure this field can add: a wrong single key drops an aged post as stale, so compare it against regex before trusting either), chart + pos (the settled reading), reason (ambiguous / absent / unresolvable / unavailable / wrong_chart; wrong_chart (#3916) = the title is absent on the one Billboard flagship the claim names and present on the OTHER flagship, and the wire does not name the claimed chart: an album called a song or the reverse, the Showgirl “No. 1 on the Billboard Hot 100” of 2026-10-04; the compose drops it as music_news_filtered reason=wrong_chart), is_debut (the DEBUT verdict off chart_presence for a live platform chart: false = established title, true = genuinely new, null = weekly chart / unknown title / memory down -> UNKNOWN, and since 2026-09-01 unknown no longer means the wire’s framing stands: the compose’s debut guard asks Billboard’s artist chart-history page and reports the bare position when that cannot confirm a debut either, see music_news_debut_framing) – the newsroom SETTLED a chart claim against the live chart before web research (#1890). The is_debut verdict stops a wire’s loose “debuts at #N” being relayed for an established title (ICEMAN by Drake, days on the Apple albums chart, shipped as “debuts at #2”): _settle_debut_framing reads the shared memory and, on false, hands the compose established_framing_note so a climb is told as a climb. charts covers EVERY chart we can read since #2258 (owner steer 2026-08-10, “should work like that for every chart”): the kworb platform dailies, all 21 Billboard charts, and the ten Mediabase radio panels (keys radio_*, read off the same hits.radio_building() document the desk’s radio lane uses). The table held 7 charts before, while the process already fetched ~38, so a claim about any of the other 31 kept the wire’s THRESHOLD wording – live on 2026-08-10, “enters Top 20 on US Rhythmic Radio (Mediabase)” shipped for Mac Miller and Ty Dolla $ign’s “Cinderella” against a real #19, and “Top 20 on US Urban Radio” shipped for Future’s “California Girls” against a real #14. WATCH the ok-rate by charts: a radio_* or Billboard key with a sustained absent means our title matching is missing that reader’s credit style, NOT that the songs fell off – those panels are weekly, so the staleness gate deliberately ignores them. A rising unavailable on radio_* is a HITS fetch problem. | reason=wire_ahead (#3824): the live WEEKLY chart is last week’s while the wire posts the new week (chartdata’s Sunday Billboard 200 lines, mn.wire_is_next_week); the reading is dropped and the claim goes to the web verify. Since #4394 a COUNT row carries detail.window (top_n = the claim’s own top-N window, whole = a whole-chart count with no window, read over the printed Billboard chart), and reason=count_mismatch = the whole-chart count is not the figure’s count, so no reading (it arms no staleness gate). reason=week_unproven = the counts agree but the post was not written in the printed week the reading carries (mn.post_in_chart_week: a Sunday or Monday post can mean next week’s unprinted chart), so no reading either.
| source_change | cogs/music_desk.py (source watcher, 2026-09-28) | surface, count (sources changed), + folded detail.sources (the names: billboard:hot100, billboard:billboard200, hits:midweek-20, hits:hits-top-50, hits:radio_adds, hits:radio_building, kworb:<page>, riaa:recent, pollstar:<chart>) — the music desk’s watcher saw one or more of its data sources publish new content, and ran the desk’s pass off-slot for every guild it may post to. Owner target: a music story reaches the room within ~5 minutes of the source changing. The watcher looks every 2 minutes; the FIRST look after a deploy is a baseline and a failed read is never a change (utils/source_watch.py). Posts it ships carry fast=true on music_desk_posted. Since #3823 the pass runs at any hour (every watched source is a numbers source); outside the calendar span the commentary lane (ktt2_chatter) composes nothing, so a fast post overnight is expected and never a chatter post. Read it on the Desks dashboard (source_changes + fast_posts). The alarm is silence: a Billboard flip week (Tuesday) with no billboard:* row, or a week with no riaa:recent / hits:* row, means the watcher or that source’s read stopped. A pass failure emits error source=music_desk_source_watch. |
| music_news_filtered | cogs/music_news.py | guild_id, phase (story kind), reason (release_age = the 2026-10-20 release-age cut on a chart story, utils/recency.py: below No. 10 and not a debut, or No. 6-10 / a debut whose title is over two years old or undated; detail.tier deep | old | no_date, detail.pos, detail.chart; a no_date cut is NOT stamped seen; on the release_age_cut panel. not_a_story = the classifier’s settled no-story verdict, stamped seen; title_ungrounded = the classifier named a work the post never names (#3824, the GUTS case: mn.title_grounded_in_wire), stamped seen; chart_ungrounded = an RSS feed item’s story names a chart its headline does not name (#3834, _feed_chart_ungrounded: Chart Beat’s “The Life of a Showgirl Returns to No. 1”, a Billboard 200 album, shipped as “No. 1 on the Billboard Hot 100” on 2026-10-04), stamped seen; X posts are not checked, because an authority often names no chart and the classifier’s chart is then right; wrong_chart = the settle read the claimed Billboard flagship as absent and found the title on the other flagship, and the wire never named the claimed chart (#3916, MusicNews._live_chart_position): the chart cannot be confirmed, so the story is dropped and stamped seen, detail.chart = the chart the claim named; replayed on the 30 days to 2026-10-06, it fires on the Showgirl shape only; off_genre = the new-release lane found a watched artist’s in-window album outside pop, rap / hip-hop and R&B, detail.genre = Apple’s primaryGenreName, the house policy’s exact-tag test, stamped seen so it emits once per guild, 2026-10-02; shortlist_full = the compose shortlist filled with count candidates still unclassified behind it – the newsroom’s one formerly silent loss, #2800 line D; ladder_rung/market_unread/market_longshot/refuted/unverified_wire/chart_cut/cert_reported/milestone_reported/chart_debut_reported/chart_repost/novelty_distribution/lane_dark/charting/nochart_unverified; cert_batch_deferred = an act with more RIAA awards in the 3-day window than its cards per act per slot share: matchup the act, count the awards waiting for a later slot (not stamped seen; they age out of the window), detail.per_act the knob, detail.headline the act’s best; owner call 2026-09-03: diversity across acts, not a cap), the desk’s plaques board lists them all; not emitted when nothing is left to fold; cert_act_cooldown (2026-09-05) = the act took a cert card inside cert_act_cooldown_hours (detail.cooldown_hours, the /menu knob, default 24, 0 = off) so it sits this slot out: matchup the act, count the awards held (not stamped seen), detail.headline its best. The per-slot share alone let a catalog batch take one card of EVERY slot: Nickelback, 2026-09-04, 15 awards, one card composed an hour for twelve hours, five reached the room and six were stopped only by the text dedup. Watch it next to cert_batch_deferred: an act should appear under cert_batch_deferred once, then under cert_act_cooldown for the rest of the day; the same act under cert_batch_deferred every slot means the cooldown came unwired); cert_batch_capped (2026-09-17) = the act’s window awards already carry cert_cards_per_batch seen stamps (detail.per_batch, the /menu knob, default 5, 0 = off; detail.carded the stamps) so the batch takes no more cards whatever the pacing: matchup the act, count the awards cut (not stamped seen), detail.headline the best of the cut. The two pacing reasons never END a batch: Skillet, 2026-09-16, 11 awards in a day, six cards in five hours at 3 a slot and a 1h cooldown, the rest still queued. Watch it as the terminal state of a batch: an act should move from cert_batch_deferred (and cert_act_cooldown) to cert_batch_capped and stay there until the window drops the batch; a cert_batch_capped act that still shows music_news_posted cards means the count came unwired), matchup, source, + folded detail (claim, note, family, key, pos, prev, chart) — a wire numbers story was DROPPED before compose. The three no_chart reasons (phase=no_chart, #top-100-debuts-no-chart) fence the one lane that states a NEGATIVE about a real act — a tracked artist’s named release that missed a chart. reason=lane_dark = the guild has not opted the lane in: its own music_news_nochart experiment is OFF (the default), so the story is dropped and marked seen. A steady stream here is the lane being dark as designed, not a fault; it goes to zero for a guild once a mod flips the experiment on. reason=charting = the VETO fired — we read the named Billboard chart first-party and the release IS on it, so the “did not chart” claim is FALSE and nothing ships (detail.pos/detail.chart name where it actually charted). This is the “prefer absent over invented” fail direction inverted for a negative claim: a false miss is the fabrication to fear, and a first-party read that finds the song kills the story. A hit here is the veto working; marked seen. reason=nochart_unverified = the lane is FAIL-CLOSED — a no_chart story ships ONLY when the independent research CONFIRMS the miss (verify_music_claim verified=true), never on source trust like a positive number, because you cannot prove an absence on trust. NOT marked seen (the tracking week is still closing, so the research may confirm it a slot later). Watch the three together as the lane’s whole funnel: lane_dark should dominate until a guild opts in, then nochart_unverified should be the main drop (most feed “flop” chatter is an insult with no verifiable miss) and a charting hit means the veto caught a wrong claim. reason=unverified_wire (#2040) means the story came from a music WIRE — an account that REPOSTS numbers rather than measuring them (may_relay_number false) — and the research did not confirm the figure, so it is dropped rather than relayed. That is the standing rule for a wire’s hard number: verify-or-drop. It is NOT marked seen, unlike a refutation, because “the web has not caught up yet” is a state that changes and a wire often posts a number ahead of the outlets. An AUTHORITY’s unconfirmed number is a different case and still ships — she measures it first-party. Watch it as a share of the wire count in music_news_slate: a rate near 100% means the widened poll is spending verify budget on accounts whose claims the web never carries, and the account list should shrink rather than the rule loosen. reason=refuted (#2035) means the verify judge reported a DIFFERENT number for this claim, so nothing ships. The judge verdict used to be one boolean and the desk read every false as “we could not corroborate it”, which RELAYS the wire’s number in her own voice; that is right for a claim the research does not cover and wrong for one the research CORRECTS. Live: the wire said Drake had 6 songs in the top 11 single-week stream list, the judge answered “7 distinct songs, not 6”, and the post shipped reading 6 on Discord and on X. The self-gate cannot catch the class — it grades the take against the claim, and the claim is the wrong number. A hit here is the guard working. Watch it as a SHARE of music_news_verified verified=false, not on its own: only the contradicted subset drops (~1 in 4 of the false verdicts, by a 30-day read; the rest are “no corroboration found” or “that figure is a projection” and still ship). A refuted rate climbing toward the whole false population means the judge is calling missing corroboration a refutation, and the desk is going quiet instead of getting accurate. A refute a week behind a number authority does not drop (#3827): when the judge’s note puts the title at the wire’s own LAST-WEEK position (the post’s “#1(+20)” against research at #21, mn.refute_is_last_week), the research is older than the post. The verdict becomes unconfirmed, the story is not marked seen, the stale research and X context are withheld from the compose, and music_news_recover fires with source, detail.via=stale_refute + detail.note (panel newsroom_refutes, beside the refuted count per source). Live cases: chartdata’s Taylor Swift and Ella Langley lines of 2026-10-04. reason=ladder_rung = the wire quoted ONE RUNG of a Kalshi scalar ladder and we could not re-read the market ourselves to replace it with the ladder’s forecast (utils/kalshi_ladder.py); compare against market_filtered reason=ladder_rung on the same tickers. reason=chart_cut (#2127) = the story is about a chart the owner cut from the newsroom’s scope (music_news.CUT_PATTERNS); detail.family names which one. The newsroom was the only chart surface with no chart list, so it relayed whatever chart a trusted account posted about — 135 chart stories over 30 days named ~20 chart families, including ARIA, the U.K. Official Albums Chart and a K-pop fan tally. Watch detail.family=unnamed separately from the named families: a named family is the cut working as asked, but unnamed means the gate could not recognize the chart at all and failed CLOSED, so a real chart appearing there is a missing pattern, not a cut. It measured 1 of 135 (a preview-truncation artifact) before shipping; a rate climbing past a few percent means the wires moved to wording the patterns do not cover, and the desk is going quiet rather than getting narrower. Re-measure with scripts/dryrun_chart_filter.py. reason=deep_chart (owner cut 2026-08-07) = a PLATFORM-chart (Apple Music / Spotify / iTunes daily) position below the top-MUSIC_NEWS_CHART_POS_FLOOR (default 10) with no debut call in the wire wording, OR a debut call past the artist’s tier debut floor (music_news.deep_platform_cut with artist_watch.debut_depth_floor: none for WATCHED, top 50 KNOWN, top 25 UNKNOWN – added 2026-09-08 after a #199 Spotify “debut” by Ashe shipped on the unconditional carve-out; a re-entry is not a debut). detail.pos is the position tested, detail.settled says whether it was the live reading or the claim’s own number (kworb down), detail.debut whether the wire called a debut (detail.event the resolved chart event and via how it was read: classifier = the chart_event field inside its bound, bounded = the field named a debut or re-entry the wire’s words do not carry, so the regex decided, regex = no field, #3418; the chart_repost row carries the same via + detail.event for its re-entry read) and detail.tier the artist’s recognition tier (a cut with debut=true is the tier floor firing; tier=known on every row means the recognition lists are empty and the floor relaxed to 50). Measured cause: popbase’s deep-chart chatter about one song shipped as hero cards two days running (#148, then #125). The desk’s watch lane is the intended supply for top-of-chart moves. A hit here is the floor working; a SUSTAINED zero alongside deep positions in music_news_posted previews means the gate came unwired. reason=chart_repost (#chart-repost) = the desk ALREADY reported this song’s position on this WEEKLY chart and it has not made a real move, so the re-report is dropped before the verify spend. A weekly chart (a radio Mediabase panel, a Billboard weekly) holds a position for the whole week, and the text dedup cannot hold it — Drake’s “Shabang” shipped at #1 on the US Rhythmic radio chart on Aug 15, 16, 19 AND 24, and Cardi B’s “AH HA” shipped at #18 then #14 (four spots) on the same chart. The high-water store (kv_cache namespace music_news_chart_hw, keyed music_news.chart_position_key = folded subject + pos + the chart key, per guild, 45-day INACTIVITY TTL — a suppressed repeat refreshes the row, so a 10–20-week #1 run cannot expire mid-streak and re-ship) holds the last position REPORTED; music_news.should_repost_chart_position re-reports only a CLIMB of MUSIC_NEWS_CHART_REPOST_MIN_CLIMB (default 5) spots, a NEW #1 (a crown the last report did not hold), or a RE-ENTRY (the song left the chart and came back — the owner steer “exits post right” — detected off the original wire wording via music_news.claims_reentry, since the settled rewrite strips “re-enters”). A plain FALL to a worse-or-equal position with no re-entry cue stays suppressed like a hold. detail.pos is the settled position, detail.prev the last reported one, detail.chart the chart key. It gates a SETTLED reading on ANY known chart — WEEKLY panels AND live DAILIES, because a held #1 repeats on both (Ariana Grande’s “hate that i made you love me” shipped ELEVEN times holding #1 on the global Spotify daily chart); the deep_chart floor runs FIRST and trims a deep daily position, so only a top-of-chart daily position reaches this gate. A genuinely new chart or a big jump still ships. Written on PRODUCTION delivery only, so a staging audition never raises the mark. Dropped BEFORE the compose spend, and the post is marked seen. A hit here is the gate working; a sustained zero alongside repeated same-song positions in music_news_posted previews means it came unwired. | reason=cert_reported (#2153) means the desk ALREADY reported this exact certification: a certification settles, so it is news once, and the text dedup cannot hold that (its memory is bounded by rows, a 36h window and 72h retention, while the WAP Diamond repeat arrived 113h later from a third account). The folded key is the music_news.cert_story_key identity (the folded subject + the model’s cert_id, <multiplier>x<level>|<body>) matched against the music_news_cert rows in post_dedup_history. Like the milestone rung, the certification identity comes from the classifier, not a regex over the claim: cert_id (e.g. 7xplatinum|riaa) so a reworded repeat keys the same, empty when the post is not a clean certification. Dropped BEFORE the research spend, and the post is marked seen. reason=milestone_reported is the SAME fix for a STREAMING MILESTONE (a track crossing a fixed stream rung): a crossing settles once, and the text window could not hold it — the J. Cole “No Role Modelz” 3B card posted twice ~54h apart, past the 36h window, because the topic judge overturned a shape-only hit against a DIFFERENT milestone while the real twin sat out of window. The folded key is the music_news.milestone_story_key identity (the folded subject plus the model’s milestone_id) matched against the music_news_milestone rows in post_dedup_history (1-year retention). The rung comes from the MODEL, not a regex over the claim prose: classify_music_news already reads every post and returns milestone_id — the canonical value|platform|metric (e.g. 3000000000|spotify|streams) — so a reworded repeat (“3B” and “3 billion”, “streams on Spotify” and “Spotify streams”) keys the same. The ONE metric held is a cumulative all-time STREAM count, and a wire’s “plays” folds to “streams” so the same crossing keys the same whichever word the wire used. The model returns an EMPTY id for a figure that must not dedup — a year-to-date/annual/period total, a territory-scoped figure, an aggregate across platforms, a YouTube or other VIEW count (per-video, not per song), or a monthly-listener figure (a moving snapshot that rises and falls) — and a throwback (retro) also fails open (#2353 review). The key builder only validates the id’s shape and folds the subject; a regex over the prose could not hold the wording variance (grouped digits, hyphenated qualifiers, post-metric platforms, territories) and each broke a different branch (#2352). Dropped BEFORE the research spend, and the post is marked seen. reason=chart_debut_reported (#market-dedups) is the SAME fix for a WEEKLY-chart DEBUT, and the FIRST cross-surface one: a song enters a chart once, so “BbY WOW debuts at #73 on the Billboard Hot 100” is not news again days later — but it shipped from music_desk on Aug 19 and from the newsroom on Aug 22 (+53h) and Aug 23 (+85h), each past the ~14h effective wire window, and the topic judge even overturned a shape-only hit against a different Karol G debut still in the window. The folded key is the music_news.chart_debut_story_key identity (the folded subject + debut + the chart key) matched against the SHARED wire_chart_debut rows in post_dedup_history (1-year retention). It is shared because music_desk reads Billboard first-party while the newsroom relays the same debut off @billboardcharts — both surfaces write the key (on PRODUCTION delivery only, so a staging audition never suppresses a real post) and both read it. Only a DEBUT keys (chart_debut_story_key returns None for a re-entry or a move, and for a LIVE daily chart that churns), so a jump (“surges to #47”) still ships. music_desk runs the same key: it emits music_desk_scored shipped=false reason=chart_debut_reported when it suppresses a debut a wire surface already posted. Dropped BEFORE the compose spend, and the post is marked seen. reason=novelty_distribution means a novelty story was a per-work DISTRIBUTION, not a single unusual count, so it is dropped before the verify spend (music_news.novelty_is_distribution). A novelty is one number (“Kid Cudi hums 1,814 times across his albums”); a per-album breakdown has no single headline figure, so the classifier put a RANGE in the card’s hero slot (“47.0%—89.4%”) and a comma list in the claim, and the card heroed a meaningless range over a caption that overflowed. Live: “Kanye West’s lyric-credit share by album runs from 47% on DONDA all the way to 89.4% on 808s & Heartbreak” (2026-08-15, @hiphopnumbers, X-only). The two upstream gates missed it — the source is a trusted relay, so may_relay was true and verify-or-drop did not fire, and the slop self-gate passed it at 0.0. The guard fires on a range figure OR a claim listing three-plus percentages, and the post is marked seen. A SUSTAINED zero here is fine — the shape is rare; a rate climbing means the wires are posting more per-work breakdowns and the novelty lane is spending drops on them. music_news_verified via (#3416): judge_count / regex_count when the card hero was a wire RUNG swapped for the research’s count (Rod Wave’s “100 entries” -> “107 career entries”), by the verify judge’s own threshold_passed + count or by the regex pair; absent otherwise. A judge_count share at zero says the judge never sets the flag; a regex_count share that stays high says it sets it too rarely. music_news_verified card_headline (#2825, folded): on a market story with no hero figure, the rung of _market_card_copy that names the card headline: award (the prize read off the claim), outcome (the classifier’s short name), act_fallback (neither, so the ACT is the title and the clause is the caption), none (no clause). Absent on every other story. The fallback is correct but raises nothing, so this is its only record. Query: where event == 'music_news_verified' and phase == 'market' | summarize count() by tostring(parse_json(detail)['card_headline']). Panel: newsroom_market_headline. No monitor: about 20 market stories reach the event in 30 days (19 with no figure, measured 2026-10-25), too few for a rate.
| music_news_slate | cogs/music_news.py | guild_id, count (the slate size), + folded detail: authority + culture (how many of each source kind the slate carries), authority_pool + culture_pool (the fresh, cued, unseen candidates each kind offered), artist_pull (proactive watched-artist pull candidates — see artist_news), new_releases (deterministic Apple album-drop candidates, leading the slate — see artist_news detail.mode=newrelease) — the per-slot CLASSIFY SLATE the newsroom built (#2031). The desk polls the whole music wire list (38 handles) rather than the 13 numbers authorities, because a chart stat only an aggregator carried could not otherwise reach it, and _MAX_CLASSIFY (12) is a PER-KIND budget so neither kind can take the other’s places. Either count persistently short of its budget while its pool is healthy means the split broke. culture_pool is the pool BEFORE the cue-strength filter, so a healthy culture_pool with culture at 0 means chart_cue_strength rejects everything, while both at 0 means the wire fetch is not reaching those handles (separable against curator_fetch). Watch culture against music_news_scored: the classify loop stops at _MAX_COMPOSE_ATTEMPTS (6) stories and the authorities lead, so a slate carrying music wires that never produce a scored story means the authorities are filling the shortlist first — measured at 6% of slots, and a materially higher rate is the signal to reserve compose places per kind. detail.rank + detail.moved (#3448): who ordered the culture tier. model = one Haiku call (claude_api purpose=music_news_slate_rank) ranked the first 40 posts of the cue order and returned indexes into them; cues = the call or the reply failed, so the cue order stood (the order before #3448); skip = the pool fit the 12-place budget, so no call was made. moved = the ranked places the cue order alone would not have given. Panel newsroom_slate_rank. No monitor: a cues slot costs ranking quality, not a post. |
| apple_releases_fetch | utils/apple_releases.py (top_albums, #release-board) | The releases board’s source: Apple Music US Top Albums, 100 rows ranked by current plays, each carrying its own release_date, in ONE keyless request. source=apple_releases, ok, count (rows parsed), duration_ms, error (fetch_failed after the retry layer | no_albums for an empty or changed body — an empty Top Albums feed is a failed read, not a real “nothing charting”). WHY IT EXISTS: the board read Deezer’s all-genres chart alone, that chart is LONG-TAIL, and the board went fully DARK — 35 runs between 2026-08-27 and 2026-09-04, 0 posts, every one phase=source_thin. On 2026-09-04 Deezer’s top 100 held three albums released that day and missed Beyoncé’s B’DAY (20th ANNIVERSARY DELUXE EDITION); Apple held four, Beyoncé’s among them. Fail-open like Deezer: an ok=False raises nothing (the board just reads fewer drops and skips the slot as a quiet night), so this rate is the ONLY sign the feed died — it feeds misc_health["apple_releases"] → the integration_unhealthy finding. On the outbound_calls and data_sources panels. All fields are shared vocabulary — zero new field names. |
| spotify_fetch | utils/spotify.py | LIVE again as action=search_artist (#3678, the first-week lane; since 2026-10-02 every StreamTotals board, as the artist-id fallback when kworb’s leaderboard misses an act; measured 2026-10-02: 658 ok, 13 fetch_failed over 7 days). Also action=search_release (2026-10-20): the release-date resolver’s LAST rung (utils.release_age._spotify_release_date), a fielded /search that runs only when MusicBrainz, Deezer and iTunes all missed; count = dated candidates, ok=True reason=no_match when none; its own action, so its long-429 block is its own. Also action=artist_albums and action=album_tracks (#3897): the first-week Songstats fill reads an act’s albums list and an album’s tracklist through _api_get. A failed call now carries http_status. reason=rate_limited (#3896 follow-up, 2026-10-06): Spotify answered artist_albums with 429 QUOTA_EXCEEDED and Retry-After 26542 (about 7.4 h) after the fill read the albums list again on every 20-minute pass. A 429 whose Retry-After is over 60 s now blocks that action in the process until the time passes; each skipped call emits ok=False reason=rate_limited with folded detail.retry_after_seconds and makes no request. It counts against the integration-health rate, so a long lock flags. action=search_artist honours the same block since #3920. It had its own request code, so retry_http retried a long 429 and the next call sent a request again (2 fetch_failed at 16:30 UTC on 2026-10-06, next to the artist_albums lock). It now goes through _api_get and keeps its shape: count 1/0, ok=True reason=no_match when no exact name matched. cache_hit=True (#3934): artist_albums (24 h) and album_tracks (7 days) are kept in kv_cache, and a read answered there emits ok=True cache_hit=True count with NO duration_ms, so it stays out of the latency tables. A live read carries no cache_hit field, so the hit rate is countif(cache_hit == true) / count() per action. Each action’s long-429 block is stored too (kv_cache namespace spotify_block), so after a deploy the first call emits reason=rate_limited with no request instead of one 429. A DB error on that row emits emit_error(source=spotify_block, recoverable=True) with context.op load | store. The ops-monitor’s misc_health["spotify"] branch skips a cache_hit event, like the other integration branches: a hit sent no request, so counting it as ok would hide real failures. Dashboard: the media dashboard’s spotify_reads table shows, per action and hour, calls, cache_hits, live (calls minus hits), rate_limited and failed. No separate Axiom monitor: a lock or an outage already lowers the spotify integration-health rate, which the ops monitor flags as integration_unhealthy. The rest of this row is the new_releases history — the releases board moved off Spotify (/browse/new-releases is 403 for a new app) to Apple’s Top Albums feed (see apple_releases_fetch), so nothing imports the client and nothing emits this event; re-measured 2026-09-04 against the live credentials — the token call returns 200, the browse call 403; utils/spotify.py + this row await removal in a follow-up. Historical shape: ok, duration_ms, count (releases returned), action (new_releases), reason (on ok=False: unprovisioned = no keys | auth = creds present, token refused | fetch_failed = HTTP error) — one call to Spotify’s Web API /browse/new-releases, the releases board’s source (client-credentials auth). Fail-open: ok=False count=0 on any of those, and a real failure also emits emit_error(source=spotify\|spotify_token, recoverable=True). Joins the ops-monitor latency read AND its integration-health read (a dedicated branch feeds misc_health["spotify"] → the integration_unhealthy finding) — but reason=unprovisioned is EXCLUDED there, so a dark-by-default install (no SPOTIFY_CLIENT_ID/SECRET) is not perpetually flagged; a creds-present auth/fetch_failed rate over the window IS a real upstream/quota issue and flags. This is the ONLY sign of a silent Spotify outage, since a thin board is logged as source_thin (out of the quality/decline findings). |
| music_releases_scored / music_releases_posted / music_releases_dedup | cogs/music_releases.py (#release-board) | The NEW-RELEASES BOARD — a scheduled roundup of the night’s fresh drops (artist - title, one card), the roundup sibling of the music DROP. Source: apple_releases_fetch – the board reads Apple Music’s US Top Albums feed and keeps the records that dropped in the window. It reads NO Deezer (owner steer 2026-09-05): Deezer’s chart alone left the surface dark for its whole recorded life (35 runs, 0 posts, every one source_thin – long-tail, missed the day’s biggest drops), and paired with Apple (#2950) its rows were the noise on the card; deezer_fetch and deezer.recent_releases were removed with it. (#2447; the Spotify /browse/new-releases source is 403 for this app, and the MusicBrainz date feed had no popularity signal so it was noise-or-thin.) music_releases_scored: guild_id, score, reason, shipped (bool >= the shared music-desk 0.6 floor – the board self-gates on music_desk_score(kind="release"), the numberless release rubric, so the ops-monitor’s 0.6 read is honest), count (fresh drops found, capped at the card’s 10 rows), category (new music friday | drops today for the board; biggest drops for the Friday-only TOP DROPS card — the two units one slot yields since #2956, each with its own floor, roster-dedup key and source_thin skip, so a per-unit read is a category filter), post_preview. A thin/empty skip rides this event with a marker phase + reason: phase=source_thin reason=thin (below the plan’s floor, NO compose attempted – a source/volume skip, kept out of BOTH the quality average and the compose-decline count, since the model never ran; an Apple outage is the apple_releases_fetch event’s job) vs phase=declined reason=empty (compose ran and the model returned EMPTY – a real compose decline the decline-rate finding counts). is_judge_score drops both 0.0s from the quality average. music_releases_posted: guild_id, channel_count (0 on a staged audition), delivered (room | staged), count, category, post_preview. music_releases_dedup: guild_id, channel_id, decision (similarity_gate), signal, post_preview. Since #4350 a text-dedup drop on the board is read again by Haiku (claude_api purpose=release_take_new): a take that names a new release ships and emits dedup_overturned surface=music_releases instead of this event, and the desk_repeat_gate panel shows both. A rise in claude_api purpose=release_take_new ok=false means the check is down, which keeps every drop (the behavior before #4350). On the shared desk_quality/desk_posts panels (it is in _DESK_SCORED/_DESK_POSTED) and in ops_monitor.SCORED_KINDS – and because it ships at the 0.6 desk floor, the monitor’s 0.6 SCORE_FLOOR reads it honestly (no false surface_dark from sub-0.6 shipped boards). All fields are shared vocabulary — zero new field names. No dedicated MONITOR: it ships dark (the music_releases experiment defaults to STAGING) and the shared self-gate + decline-rate findings already cover it. The FIRST-WEEK card (category first week, 2026-09-22) rides the same events: music_releases_posted on delivery, and music_releases_scored ONLY for its source_thin skip (fewer than 2 drops resolved). Its line is built from the card’s top row, so no judge runs and a built card emits no score. Its per-drop stream reads emit album_total with surface=music_releases, phase=page (reason no_artist_id / no_match / no_kworb_albums on a miss), plus phase=first_week reason no_daily for a matched row whose Daily cell is blank, which the existing album_total panel already splits by surface. The Songstats fill (#3896) adds album_total phase=songstats: ok=True, source=songstats for an album summed from its tracks, else reason no_album (Spotify has no matching release within 2 days) | no_tracks | rate_limited (#3920: a Spotify long-429 block made the albums or tracklist read return [] with no request; before, a blocked albums read logged no_album, 20 such rows after the #3915 deploy; a blocked read is not kept as a miss) | pending (the pass ran out of calls; the next pass continues from the cache) | incomplete (a track had no history or lagged a day). Its Spotify reads emit spotify_fetch with action=artist_albums | album_tracks. Its staged card rides music_releases_posted delivered=staged category="first week". No new field names. No monitor: a miss leaves one row off one card, and the panel’s phase=songstats split shows the hit rate; Songstats’ own budget is already on the quota panel. Reissues off the card (#3929): the first-week card leaves a reissue off its roster and emits album_total phase=first_week reason reissue for it. Since #3445 a reissue is a title _REISSUE_RE matches, OR a title carrying Deluxe / Edition / Version / Live / Expanded that the cached Haiku check claude.release_is_reissue (purpose=release_reissue, labels REISSUE / NEW) calls a reissue or gives no verdict on (the stat is omitted on a model error). The same check gates the board’s first-night stat and the Songstats fill. The calls show in claude_api with purpose release_reissue; a rise in ok=false there means more stats are omitted. The first-week roster also reads the release-version verdicts now (#4230 item 3), as the board does. The release_board_songstats experiment defaults to PRODUCTION since #3929. No new field names. No new panel: the album_total panel splits by phase and reason. No monitor: a skipped reissue is the intended outcome, not a failure. House genres on the board (#3276): the board, the Friday top-drops card, the first-week card and the per-album boards keep rap, R&B and pop only (release_board.house_rows, Apple genre ids through music_policy.in_house_genre_ids). Each in-window album cut emits album_total phase=albums (the board) or phase=first_week, reason off_genre, query artist - title. A Kalshi forecast album takes the genre of its Apple row; with no Apple row it has no genre data and stays only when the act is on the kworb watch list, else album_total phase=first_week reason=no_genre (a kworb outage counts as not known here: the list stands in for the missing genre). One event per cut row per slot, so a row in the window emits a few times a day. No new field names, no new panel (the album_total panel splits by phase and reason), no monitor: a cut is the intended outcome. A rising no_genre count with kworb healthy means the forecast albums stopped matching Apple’s titles. First-week total + per-album boards (#3935): each fill pass emits album_total phase=week: ok when the album has a clean first-week baseline, else reason no_baseline (the row stays on the card without its wk figure); and phase=daily: ok when every track has a clean one-day figure, else reason not_clean (the row stays with no daily). The card emits album_total phase=first_week reason=outside_window for a drop whose Spotify release date is more than 7 days old or in the future. The per-album boards emit album_card with surface=music_releases (ok + result_count = tracks on a build; reason not_ready when a due album’s Songstats tracks are not complete or fewer than half have a clean one-day figure) and ride music_releases_posted category="album board" on delivery. No new field names. The album_card panel now splits by surface. No monitor: a miss costs one figure or one staged board, and the phase=week and album_card ok rates on the panels show a sustained drop. |
| watch_new_entry | cogs/music_desk.py | ok (the row was datable), reason (debut | reentry | charted | no_date | deep | radio_entry | side_version), + folded detail: chart, pos, age_days, last_seen_days, title, judge — the watch lane’s new-entry gate (#2045). side_version (#3441, 2026-10-16) is a live / remix / sped-up re-cut. The regex (release_type.is_side_version) screens first. A title it flags goes to the cached Haiku release-version check (claude.release_version_kind, purpose=release_version, labels new / recut / edition) before it drops; judge says which one decided (model or regex, the regex when the model gave no verdict). Since #4230 every side_version drop carries judge, including the model’s recut call on a regex-passed title (judge: model), so judge == 'model' counts all of the model’s drops. Since #4708 a title whose version suffix names a rendition (release_type.rendition_marker: instrumental, a cappella, karaoke, sped up, slowed, reverb, live, acoustic, edit) logs judge: rule, because release_version_kind returns recut for it with no model call. Before #4708 those drops logged model, so judge == 'model' counted calls that did not happen. A title the regex passes is asked only when the lane is about to KEEP it (debut, reentry, radio_entry), so the charted / deep / undated rows cost no call. A model-dropped kept row emits side_version with age_days set. The calls show in claude_api with purpose release_version; the release board asks the same check about its first 20 in-window rows (cogs/music_releases.py:_version_verdicts). The album-milestone count asks claude.album_variant_relation (purpose=album_variant) about every related title pair since CB-17 (about 3-5 calls a crossing, cached 14 days); the per-track card’s album join asks claude.track_version_join (purpose=track_version_join) about a row that joins a track only by the loose key (CB-24, under 1 a day). kworb marks a catalog RE-ENTRY NEW exactly like a fresh drop on the rank-only charts, so chart_stories emits an unresolved new_entry and this dates the release before anything is claimed. The third lane to carry this gate, after apple_debut (#1887) and called_shot (#2038) — the same one bug in three places. charted is the CHART-PRESENCE rejection (utils/chart_presence.py, first-sighting semantics per #2139): the title is past its 24h first-sighting window on this chart (last_seen_days = days absent before its latest return; 0 = continuously charting), so kworb’s NEW flag is a tracked-range bounce or a re-surfacing recent release, and the date gate alone would have re-claimed a debut — the duplicate-debut fix (a week-old release’s age passes the window every time it bounces back). A charted row makes no post. Unlike the other two lanes, a reentry here is KEPT and posts as a re-entry: a catalog track returning to a platform chart after a real absence is real news (it is what the wires post as “‘ICEMAN’ has returned to being the #1 Hip-Hop album”), it just is not a debut. deep is a settled re-entry the DEPTH then cut: it landed below MUSIC_DESK_WATCH_REENTRY_MAX_POS (10). A new title is news wherever it lands, but an older one returning deep in a chart is churn — measured over the 14 days to 2026-08-08, all 14 re-entries the lane kept sat at #56–#99 and none reached the top 10, and the #97 one (“Where We Go”, Marshmello & Thomas Rhett, US iTunes) shipped to X. music_news.deep_platform_cut already cut this exact shape at the same depth for the wire lane; the desk lane had no floor. Debuts are exempt. radio_entry (2026-08-17) is a new_entry on a LEADING chart — radio airplay (_LEADING_CHARTS) — KEPT as a position rather than a debut. Radio builds a song’s spins over WEEKS after release, so a first appearance on the airplay chart is decoupled from the release date, and the release-age gate cannot tell a genuine debut from a song radio picked up late. A 2-week-old song entering radio at #72 passed the 45-day window and shipped as “‘petal’ debuts on the US radio-airplay chart at #72” (owner call). So the lane makes NO release claim for radio: it states only WHERE the song sits (“now on the US radio-airplay chart at #N”, kind radio_entry), and only when the entry lands inside MUSIC_DESK_WATCH_RADIO_ENTRY_MAX_POS (20, anchored to the per-format lane’s top-15 breakout cap). A deeper entry is deep-tail airplay churn and drops as deep (petal #72); a tracked-range bounce drops as charted; no release date is ever read for radio. The crown / top-10 / jump kinds carry no release claim and are untouched, and utils/radio.py still tells the “climbing on radio” story off the spins. So reentry + charted + deep + radio_entry streams are the gate working. Watch instead for a sustained no_date stream: that is Deezer failing, and the lane’s new entries are dark. The PRE-ORDER rule (#3483, 2026-09-19): an established title RELEASED within MUSIC_DESK_PREORDER_GRACE_DAYS (1) resolves debut, not charted — a title cannot be established before it exists, so the earlier sightings were its pre-order listing (“Bass Persuades” at #15 on Apple Music albums on release day, first seen 12 days before, rejected every slot). A row that states its own release date (the iTunes albums reader) is dated from the row and pays no catalog lookup; for an ALBUM the shared resolver now takes the LATER of Deezer and iTunes, because Deezer answers a pre-order listing’s date (2026-09-03 against the real 2026-09-18). A debut with age_days 0-1 and last_seen_days set is that rule firing. |
| called_shot | cogs/music_desk.py | ok (the row is callable), reason (fresh | stale | no_date), + folded detail: rank, age_days, max_age_days (the limit the lane applied, _CALLED_SHOT_MAX_AGE_DAYS; since #4445 the ops monitor and the desk_debut_ship_rate panel read it from here), album — the HITS called-shot lane’s DEBUT gate (#2038). The projection document has no last-week position for a catalog RE-ENTRY either, so new_entries() mixes real debuts with albums that fell off the chart and came back; the album’s release date is what separates them, and the gate fails CLOSED (no_date refuses the claim). The HITS twin of apple_debut. Measured live on the building chart dated 2026-07-30: 4 of 7 no-last-week rows were re-entries — Drake’s VIEWS (2016) at #44, Taylor Swift’s FOLKLORE (2020) at #49, Pitbull’s GREATEST HITS at #50, Megan Moroney’s CLOUD 9 at #47 — and three sat inside the watch band (#2007) that reaches rank 100 for exactly those artists, so the lane could have called a ten-year-old album as a debut. Watch a sustained no_date stream with zero fresh: that is Deezer lookups failing and the lane going dark, which is otherwise indistinguishable from a quiet chart week. |
| proj_debut_gate | cogs/music_desk.py | ok (at least one row confirmed), count (rows kept), + folded detail: considered, fresh, stale, no_date, dropped — the DEBUT gate in front of the two PROJECTION BOARDS (_fresh_projection_entries, #3170). Same bug as called_shot, one lane later: projected_debut_rank and projected_debut_sales drew new_entries() raw, and that column reads the same for a catalog re-entry, so the card for the chart dated 2026-09-19 shipped nine rows under the hero “Projected Sales: New This Week” and eight were catalog — Kanye West’s GRADUATION (2007) at #32, Drake’s VIEWS (2016) at #50, Beyoncé’s B’DAY (2006) at #8, Dolly Parton’s greatest-hits comp at #21. Both boards now take the caller’s confirmed rows and are ABSENT without them (fail closed), under a floor of 2 confirmed rows. ONE event per evaluation, not one per row, and deliberately NOT in ops_monitor._DEBUT_GATE_SHIPS: the rows overlap the called-shot lane’s, so a per-row shape would double that lane’s counts, and a light release week legitimately confirms nothing — the gate_dark finding would fire on the gate working. The board’s own silence is already covered by lane_dark on phase=chart_proj, which the three RANKING boards keep alive (they make no claim about what is new and do not read this gate). Read the tally, not the count: a stale majority is the gate working (8 of 9 on the document dated 2026-09-10), a no_date majority means Deezer and iTunes stopped answering and both boards are dark for that reason. |
| apple_debut | cogs/music_alert.py | reason (fired | deep | slipped | stale | charted | no_date; slipped since 2026-10-23 = a real debut that FELL below its entry rank inside its first day, which no longer posts, owner steer “lets just do rises”; detail.slipped counts them), count (NEW chart entries evaluated), + folded detail: pos, age_days + tier (watched/known/unknown) on fired, charted (# rejected as already-charted), deep (# cut by the depth floor, on EVERY disposition), tiers (live/off — was the recognition list real?), song, chart (songs/albums) — the Apple Music DEBUT lanes’ gate decision, for the songs chart and the Top ALBUMS chart (#1887). The lane runs FOUR gates and each rejection has its own reason. kworb’s NEW marker also fires on a RE-ENTRY, so stale is the release-date rejection (every NEW entry resolved a date, all outside the fresh window) and no_date is Deezer resolving nothing — the fail-CLOSED direction, a lookup problem rather than a real “all old”. charted is the chart-presence rejection (utils/chart_presence.py, first-sighting semantics per #2139): the title is past its 24h first-sighting window on this chart, so the NEW flag is a tracked-range bounce. deep is the tiered DEPTH floor (#2207): the debut is real, and the lane cut it because it landed below the floor for its artist’s recognition tier — MUSIC_DEBUT_KNOWN_MAX_POS (50) or MUSIC_DEBUT_UNKNOWN_MAX_POS (25), with a WATCHED artist exempt. It was added after the albums lane posted, and crossposted to X, “Lil Tony Official’s ‘ELIJAH’ debuts at #98 on the Apple Music US albums chart” — the third-to-last rung of a top-100 chart by an artist on neither kworb ranking page. deep is reported ahead of stale because a depth cut always passes the date gate first, so without the ordering the cut would read as “the dates were old” when the lane made a policy choice. deep rides the fired event too, not only the terminal one: a row cut above a row that then shipped would otherwise leave no trace, and the depth panel would under-report. tiers is the health of the depth floor’s other half — off means the recognition lists came back empty, so every artist read as known and the floor silently relaxed to the looser bar (the tier_gate_off finding). A no-NEW day emits NOTHING (the quiet norm; a kworb fetch crater is caught by the chart_fetch health watch). So deep + charted + stale streams are the gates working. Watch instead for a sustained no_date stream with zero fired: that is Deezer failing and the lanes dark, which a quiet chart day is otherwise indistinguishable from — and that whole class is now a named finding (gate_dark), because no judge-score check can see it. entry + framing say WHICH WORDING SHIPPED (2026-08-09). The lane picks one of four phrasings for a fired debut and the choice turns on the ENTRY rank, which the event did not record — so a misfiring rule looked identical to an ordinary debut. framing is debut (still at its entry rank), ramp (released within MUSIC_DEBUT_RAMP_MAX_AGE_DAYS and risen since first sighting — takes the DEBUT wording, because on a realtime chart the first-sighting rank records when the bot polled, not where the title entered), climb (risen, but the release is older, so it is a real climb) or position (fallen, which is never a ramp; the blob states the current rank only and names no move – the label was slip before 2026-09-23). entry rides along so a query can re-derive the branch without trusting the label: ['tootsies'] | where event == 'apple_debut' | extend d=parse_json(detail) | summarize count() by tostring(d['framing']). Watch for ramp on a chart where releases are rare, or climb never appearing at all — either says the age threshold is mis-set. Panels: desk_debut_gates (reason mix), desk_debut_ship_rate (per lane+chart, the one that separates songs from albums) and desk_debut_depth_cuts (what the floor cut, by tier and rank). The PRE-ORDER rule (#3483, 2026-09-19): the same exception as watch_new_entry — an established title released within the grace FIRES with framing=debut and entry=0 (the pre-order sighting’s rank is not an entry), instead of counting toward charted; an iTunes albums row is dated from its own releaseDate. |
| chart_peak | cogs/music_alert.py (_peak_core) | reason (fired / lapsed / fell_unposted; since 2026-10-23 a peak the title fell from before it held does not post (owner steer “lets just do rises”), so an ended stay reports fell_unposted and lapsed no longer fires), count (candidates at a new peak this tick; for lapsed, the stays this read timed out), + folded detail: pos, chart (ap_songs / ap_albums / it_songs / it_albums), age_days, baseline (the rank it beat), told (true = the baseline is a rank this guild was already posted; false = the entry rank), entry, fell, now_pos, held_min (minutes the title held the peak), song — the NEW-PEAK lanes (2026-10-02, the FOMO miss: “HABIBTI (FOMO)” went #6 -> #2 on iTunes Top Albums on release day and nothing posted, because every climb lane reads the chart’s day-over-day move column, which is 0 on release day). A title inside the top ten (not #1, the crown’s story) fires when it stands at its best rank so far and beats the told rank. Memory: utils/chart_peak.py (kv_cache namespace chart_peak). Since 2026-10-17 (#4455, owner: “limit peak lane, stop at highest rank achieved”) a peak fires only when its climb has stopped: the title held it, unbroken and without a better rank, for _PEAK_SETTLE_MIN (270), or it fell from it (detail fell true, now_pos = the rank it stands at now; pos stays the peak). So a climb posts once at its top. lapsed is an untold top-ten peak whose fall timed out (FALL_KEEP_HOURS, 24) without a post: that top never posted. fired and lapsed are the only emits: a tick with no new peak is the norm. The post lands as music_alert_posted with phase = apple_peak / apple_album_peak / itunes_peak / itunes_album_peak. Panel chart_peak_fired; no monitor (a quiet day is normal).
| chart_crown | cogs/music_alert.py | reason (fired | unknown_act — RETIRED 2026-09-28 (owner: “top 10 should be top 10s, they’re not limited”): no crown is limited by the recognition list any more, so this reason no longer fires. Historically: a STORE-chart #1 (iTunes songs / albums) held by an act outside the recognition list, which the lane declines to card: the fanbase-push filter artist_watch.recognized_for_store_chart, 2026-09-19; the desk’s _TOP_KNOWN_CHARTS applies the same rule to its cards, so a quiet iTunes crown with this reason streaming is the filter working), + folded detail: chart (global_daily/apple_songs/apple_albums, and since #3799 the eight iTunes GENRE chart keys itunes_{hiphop,rnb,pop,kpop}_{albums,songs}), pos, prev (the rank it came from; null when there is none), age_days (None = Deezer could not date the row), framing (ramp/climb/entry/took_top/standing — standing = the change is unproven, so the blob states only that the title IS #1: on the iTunes albums chart the clean-chart seed revealed this #1; on an iTunes GENRE chart (#3799) it is the chart’s FIRST read, whose launch seed posts the current #1 on day one), song — the shared #1 TAKEOVER lane’s wording decision (#2235). Since 2026-09-28 (owner: “we can’t miss number one changes”) the lane fires on THREE signals, not one: a climb (pos_change > 0, framing climb or ramp), a title that ENTERS at #1 (is_new, framing entry; a new entry carries pos_change == 0, so the old gate dropped it), and, on the iTunes albums chart, the reader’s own #1 record (took_top, framing took_top, no prior rank stated). The event fires at DETECTION, every tick, before the per-lane gate; a crown lane now may post every DISTINCT #1 in a day (_daily_gate keeps today’s delivered keys), not only the first, and a crown that did not deliver (a take the 0.6 self-gate blocked) is tried again next tick, up to MUSIC_CROWN_MAX_TRIES (3) times per #1 per day. So repeated fired rows for one song are normal while it retries or while a held key waits; a crown that never reaches music_alert_scored shipped=true after its tries is the miss to look for. The lane fires only on pos_change > 0, so by default the title held a real prior rank and the blob says “climbs to #1, up from #N” and forbids “debut” — the #2140 fence, and this lane is where that bad post came from. The one exception is a RELEASE-DAY RAMP, the crown half of #2226’s debut-lane rule: kworb’s pages are near-realtime snapshots, so a title released today enters low and rises through the day as plays accumulate, and its earlier ranks are that ramp rather than a chart history. A #1 reached within MUSIC_DEBUT_RAMP_MAX_AGE_DAYS of release is a real #1 debut. framing says which wording shipped, because the branch turns on a release date the event would not otherwise carry — without it a misfiring ramp reads as an ordinary crown. Fail direction is CLIMB: an undatable row (age_days null) keeps the climb wording, because pos_change already proves a climb while “debut” is the claim needing evidence. Note the ramp blob never names prev — the number rides this event for diagnosis but is kept out of the compose’s context, so the model cannot cite a rank it was never given. Watch for a sustained all-ramp mix, which says the age threshold is mis-set. refreshed_at + refresh_age_s (#3191) are the DETECTION half of the lead-time pair: refreshed_at is the chart PAGE’s own refresh time (kworb’s last-modified), and refresh_age_s is how stale that was when the lane DETECTED the new #1. This event fires while the lane BUILDS the signal, before the daily gate, the self-gate and the repeat check — so a crown counted here may never have posted, and this is NOT the “how fast did users see it” number (Codex on #3193). That one rides music_alert_posted.refresh_age_s, which only a delivered post emits, and crown_lead_time reads it. Keep both: comparing them splits a slow READ (the cache) from a slow COMPOSE (the gates and the model). Both are empty/None when the client cannot report one: absent over invented, because stamping our fetch time there would read as the chart’s refresh time and be wrong by however long the cache held. The RADIO crown lane emits this event now too. It composes its own blob (an audience figure, not a rank) so it never rode _chart_crown, which left it the one crown lane with no event at all; framing is always climb there (radio adds take weeks, so a release-day ramp is impossible on that chart). Watch refresh_age_s for a creep back toward the old 3h: that means revalidation stopped working and the lanes are late again. Panels: desk_crown_framing (this event), crown_lead_time (the delivered twin on music_alert_posted). |
| chart_jump | cogs/music_alert.py | reason (standing / fired), + folded detail: chart, pos, jump (spots climbed), top_ten (the act’s top-10 count), posted (did another act’s climber take the slot), song — #2672. fired (#4460, 2026-10-17) is the lane’s PICK: one row per jump the lane offers to post (detail chart, pos, jump, song); the post itself lands as music_alert_posted with phase = the jump lane. Since #4460 every jump lane passes over a climber whose rank, or a better one, was told for that title in the last _JUMP_TOLD_DAYS (1) by any rank lane on the chart (_TOLD_LANES; the Spotify jumps record told ranks too). That refusal emits nothing. Panel chart_jump_fired; no monitor (a quiet day is normal). standing: Emitted only when a jump lane (spotify_jump, spotify_us_jump, apple_jump, itunes_jump) STEPS ASIDE because the climber’s act holds chart_boards.MIN_TOP10_STANDING (2) or more of the chart’s top 10, read with chart_boards.artist_top_ten_count, the desk’s own count. The desk’s watch_artist_board owns that story. A step-aside raises no error, so this event is its only trace. posted=false is a slot the lane left empty; a run of those for an act the desk does not board (the desk boards only recognized acts on the store and radio charts) is the fall-through the issue warned about. The crown lanes do not step aside (owner steers of 2026-09-28: every #1 change posts). |
| music_alert_posted / music_desk_posted / music_news_posted source_age_seconds (music lag) | cogs/music_alert.py, cogs/music_desk.py, cogs/music_news.py (#3829, epic #3821, 2026-10-05) | Folded detail.source_age_seconds: the seconds from the UPSTREAM change to the post. It reuses the wire desks’ field, so it adds no field name. The upstream time per lane: an alert settle and a desk reveal read Kalshi’s settlement_ts (reading.settled_ts); the alert’s crown, jump, debut, peak, radio and release lanes read the chart page’s refresh time (_LAG_LANE_PAGE in cogs/music_alert.py, one page per lane, the same reader as the lane’s crown); a desk chart_proj board reads the HITS document’s _updatedAt (the unit’s upstream_ts); the desk watch_chart cards, platform_board / board_presence boards and the kworb milestone cards (career streams, monthly listeners, a track’s or an album’s streams) read the refresh time of the kworb or iTunes page they were built from (upstream_ts, #3842, through the shared utils.break_latency.page_refreshed_at, read right after the fetch; _CHART_PAGE in cogs/music_desk.py maps a chart key to its page); the newsroom lane=x_watch reads the authority tweet’s created_at, and lane=feed_watch (#3834) reads the RSS item’s published time. A page clock is the time the page last changed, so on a page that refreshes often (Apple, iTunes) a story first seen on an earlier refresh reads younger than it is: the age is a lower bound there. Every other lane has no upstream time, so the field is absent, never 0: an alert line_move / last_call (Kalshi gives no time for a move; a settle’s close_ts fallback is unix seconds, which the age helper reads), the artist-market lanes, and the desk lanes whose source gives no honest time: chart and artist_catalog (Billboard; the client exposes no publish time), projection / tracking (a live Kalshi price has no change time), a HITS year-to-date album milestone (the client does not expose that document’s _updatedAt), and a Songstats subject milestone. Panel music_lag (Desks dashboard, p50 / p90 by event + lane). Gate: the ops-monitor music_lag_slow finding (p90 over 600s per lane at n>=5). Query: ['tootsies'] \| where event in ('music_alert_posted','music_desk_posted') \| extend age=toint(parse_json(detail)['source_age_seconds']) \| where isnotnull(age) \| summarize percentile(age,90) by event, phase |
| music_news_posted age_read | cogs/music_news.py, utils/news_age.py (age_read_text) (#4684, 2026-10-23) | Folded detail.age_read: the Grok news-age read’s prose (up to 200 chars from its start, citation links cut), which states when the EVENT happened. is_stale_recap(record=...) fills it on a FRESH or a STALE verdict; the newsroom carries it in Composed.meta to the post. Absent when the gate made no read (no Grok client, an error, no verdict). source_age_seconds dates the source TWEET; this dates the STORY, which is the gap #4671 shipped through (a fresh tweet about a June award). Read by the ops-monitor story_age_stale finding. No new top-level column. |
| competitor_match | scripts/competitor_scoreboard.py (#3862, epic #3821, 2026-10-05), run from the ops-monitor job | The competitor scoreboard: the same music story from @chartdata, @talkofthecharts, @chartmastersorg or @hitsdd against ours. Not emitted by the bot: the script ingests the rows straight into the dataset, with _time at the POST time (ours for a match, theirs for a miss), so a week groups by when the story posted. kind = match (one row per our post and competitor that tell the same story) | miss (a story of theirs that none of our posts within 12h tells) | run (one health row per scoreboard run). On match: surface (music_news | music_desk | music_alert), phase (our lane), source (the competitor handle), matchup (our subject, 80 chars), delivered (x = our clock is the X crosspost time, room = the Discord post time), + detail (a JSON string): lead_seconds (their post time minus ours; positive = we were first, negative = they were), chart (the charts_named keys, cut family or source the story names), tweet_id (theirs), via=relay (the newsroom took this story FROM that account, so it was first by construction). On miss: source, post_preview (their text, 120 chars), detail{chart, tweet_id}. On run: ok, count (competitor posts read), duration_ms, error (on a failure), detail{pages, judge (CB-28: model = the Sonnet pair read and the Haiku story read ran, rules = no key or no client, so the rule ladder decided), judge_reads, judge_errors (reads that fell back to the rules)}. The reads are claude_api events with purpose competitor_same_story and competitor_story, but they reach only the ops-monitor job log: the job never calls axiom.configure, so emit() does not send them to Axiom. The Axiom signal is this run row’s detail.judge and detail.judge_errors, which the script ingests itself. Matching reuses utils.music_news (artist_grounded_in_wire, title_grounded_in_wire, charts_named, chart_family, claim_platform, ungrounded_quoted_titles) plus a count / cert / position agreement check; the module docstring lists the rules. Field cost: none. The top-level names are registered shared ones, and the detail keys live inside one JSON string (not _FOLD_FIELDS, which polices emit() calls; this script does not call emit()). Panels: competitor_lanes + competitor_accounts (Content Desks). Gated by the ops-monitor competitor_slow finding. |
| spotify_debut | cogs/music_alert.py | reason (fired | deep | slipped | stale | no_date | side_version; slipped since 2026-10-27 = a debut below its first-sighting entry rank, cut like apple_debut’s, counted in detail.slipped), count (fresh chart entries evaluated), + folded detail: pos, age_days (on fired), entry (the first-sighting rank, 0 = unknown), days_on_chart, song, chart (global_daily), tier (on fired), tiers (live | off), deep (rows the depth floor cut). The depth floor is #2213: the shared aw.debut_depth_floor table scaled x2 for the top-200 Global Daily (WATCHED no floor, KNOWN to #100, UNKNOWN to #50), run after the date gate and skipping rather than ending the scan, as on apple_debut. deep is reported before stale because a floor cut always resolved a date. This row is the Spotify RELEASES lane’s DEBUT gate, the Spotify twin of apple_debut under the same fail-CLOSED contract. Real case: the lane selected ROA’s “VAMO A VEL” at #191 on 2026-09-22. kworb’s days-on-chart column counts days on THAT chart, never days since release: a song released weeks earlier that only now crosses into the Global Daily top 200 reads days=1, identical to a same-day drop. Shipped in prod on 2026-08-04 as “Steve Lacy’s ‘oh yeah?’ opens with 8.4M Spotify streams day one” — the song came out 2026-07-17 and had charted in the US every day since; it entered the GLOBAL chart at #126 that morning, and the 8.4M was a running total, not a first day. The lane now confirms the Deezer release date (_debut_release_age, the gate the Apple debut lanes already ran) before it calls anything a new release. Watch a sustained stale / no_date stream with zero fired: that is the lane dark, which a quiet chart day is otherwise indistinguishable from. The gate runs BEFORE the Songstats lookup, so a rejected entry costs no metered slot. entry + days_on_chart are the #2224 fix. chart_debuts accepts a row up to MUSIC_DAILY_DEBUT_MAX_DAYS (2) days on the chart and the row carries its CURRENT rank, so the blob’s “entered the chart at #N” was stating a day-2 rank as an entry rank — #2140 on the lane that never got it. Measured live: Stray Kids’ “This & That” entered at #31 on 2026-08-08 and still counted as a debut at #82 on 08-09, so a post one day later would have been wrong by 51 places; it did not ship only because _daily_gate’s same-key rule blocks a REPEAT, which is dedup and not correctness. The lane now reads chart_presence under the shared sp_global key for the entry rank, falls back to the current rank only on a day-ONE row (where the two are the same by definition), and otherwise claims no entry rank at all. The post leads with the LATEST rank — owner steer 2026-08-09, and chart_presence’s own #2139 rule that a consumer inside the window reports the CURRENT rank. A CLIMB carries the entry as context (“is at #20, up from a #31 entry inside a day”); a FALL carries only the current rank and “not its debut position”, with no entry rank and no direction (owner steer 2026-09-23). Either way the current rank is never read as a debut, which is the #2140 half. The “inside a day” phrase is gated on info.new, so it is dropped once the row is past its 24h first-sighting window. Presence is read for the RANK ONLY here, never for suppression: the Spotify days-on-chart count is cumulative so a bounce never reads as day 1, and info.new is a 24h window while this lane allows 48h, so suppressing on it would drop every legitimate day-2 debut. days_on_chart also fixes the other number in the same sentence — the plays were narrated as “its first day ON THE CHART” on a day-2 row |
| radio_debut | cogs/music_alert.py | reason (fired | crossover | no_date), count (radio entries that matched a big Spotify hit), + folded detail: pos, age_days, days (radio chart days; 1 = the entry day, #3683), song — the RADIO ENTRY lane’s age decision (#2077), the fifth #1887-class fix. Radio’s days column counts weeks on the RADIO chart, never days since release, so the lane composed “a BIG new song just DEBUTED” about whatever crossed to radio — live it called an 847-day-old Malcolm Todd track new. Unlike its siblings this is a FRAMING split, not a gate: the lane’s real picks measured BIMODAL (fresh releases reaching radio at 5–21 days, a summer single crossing at ~44–48, the 2.3-year outlier), and a months-old streaming hit finally reaching radio is a real story — arguably the better one. fired composes the NEW framing (the release age is stated in the blob); crossover composes the crossover framing with “new”/”debut” banned; no_date takes a NEUTRAL framing — both “new” AND “established hit” are banned, because an undatable song may be a drop-day release the catalogs have not indexed yet (#2125), so asserting either age is invented. Neither non-fired reason blanks the lane. The window is MUSIC_RADIO_ENTRY_NEW_MAX_AGE_DAYS (60), sized to radio’s measured lag — a streaming-lane 14d window would misframe most legitimate radio arrivals. |
| radio_standing | utils/radio.py (standing, #4172) | ok (a format row matched the song), reason (hit | no_title | no_artist), count (format lines returned) — ONE row per per-song radio lookup over a non-empty Mediabase chart (the music desk’s context join, cogs/music_desk.py). Most songs are not on radio, so a single no_title is normal; read the ok-rate and the no_title share over a day. Why it exists: #3454 made the title match strict, and Mediabase glues the feature into the track (“Jaded f/Ella Langley”), so 0 of the 45 “f/” rows on the live building chart matched and the radio clause dropped out of the desk context with no signal. Both titles now pass through title_forms.bare_title first. A sharp fall in the ok-rate, or no_title rising while the chart is full, means a new source title form the stripper does not read. no_artist = a title matched and the credit did not (a same-titled song by another act, the refusal working). Graded (#4180): the ops monitor files radio_title_blind (medium) when the no_title share is 85% or more over at least 10 lookups in the window. The baseline is the first chart lane after #4178 (2026-10-20): 33 of 66 lookups no_title (50%), 30 hit, 3 no_artist. The lane runs once a chart week, so most windows hold no rows at all. Panel radio_standing on the desks board. |
| stream_lookup | cogs/music_desk.py, cogs/music_news.py, cogs/music_alert.py | query (the release or act, <=120 chars), surface (music_desk | music_news | music_alert), source (songstats | kworb | none), ok (source is songstats or kworb) — did a scheduled music surface get USABLE streaming numbers. songstats_fetch is the HTTP layer under it and grades only the call, so a hollow payload (HTTP 200, the release’s counts not ingested yet) is songstats_fetch ok=true and stream_lookup ok=false. Graded per surface (#2687): the ops monitor files stream_lookup_hollow (medium) for a surface whose ok rate is 10% or less over at least 5 lookups and 3 distinct releases. Measured over 30 days to 2026-10-25: the newsroom 96% ok, the alert 0-100% per 12h window, the desk 40-90% on a normal day. The desk read 0% on 750 lookups over 2026-10-02/03 and 0-9% over 2026-10-23/24, while songstats_fetch stayed at 99%: both would have fired. Panel stream_lookup_by_surface on the desks board. |
| music_news_retired | cogs/music_news.py | guild_id, channel_id, reason (the dedup signal: text_similarity | content_overlap | topic_restated | shared_run), phase (story kind), source (the @account), post_preview — the newsroom RETIRED a source tweet because the repeat check dropped its composed unit (#2037). The desk marked a post handled when it posted, when the self-gate failed it, and when the classifier called it a non-story — but not when the repeat check dropped it, so the post stayed a candidate for its full 30h freshness window and re-composed every slot, paying one Sonnet write plus one judge call each time. Measured over 7 days: 892 composes covering only 95 distinct subjects — 9.4 writes per story (petal 136, AH HA 102, Music/Fashion/Film 75) against 104 posts, so any read of the funnel off the compose count was ~9x inflated. Pair with post_dedup on the same reason to confirm every mechanical/topic drop retires its source; a post_dedup stream with no matching retire means the unit carried no external_id |
| dedup_overturned | utils/dedup.py (arbitrated_match), emitted by cogs/scheduled_poster.py + cogs/discourse.py + cogs/chimein.py | surface (music_news/music_desk/… plus discourse and chimein), guild_id, channel_id, signal (the mechanical signal overturned), reason (the judge’s words), post_preview — the topic judge OVERTURNED a mechanical dedup hit, so the post shipped (#2037). The mechanical gate compares WORDING, which on a chart desk is mostly boilerplate, and its hit used to be final: live against a real 20-post history it called a different artist’s identical-shape line a repeat, and the same song at a NEW chart position a repeat. The judge now arbitrates instead of only backstopping. A healthy rate is small and non-zero. Zero means the arbitration is not running, or the judge is unreachable — which fails CLOSED and leaves the mechanical verdict standing, so an outage looks like silence here rather than a flood of repeats. A rate approaching post_dedup on the same surface means the mechanical gate is firing on shape far more than on stories, and its thresholds want a look. Always read the two together. The arbitration is shared, but the CATCH direction is not (#2083). The decision lives once in utils.dedup.arbitrated_match, which the wire desks, discourse and chime-in all call. The desks pass catch=True and pay one judge call per post, because they write from a house style and restate each other constantly. Discourse and chime-in pass catch=False, so the judge runs only when the mechanical gate has already found a hit: their gates fired 9 times and 0 times in 30 days, against 705 mechanical drops across the desks in 7, so catching there would buy a handful of suppressions for thousands of calls. For those two surfaces the DROP side is not post_dedup — read discourse_dedup and chimein_evaluated decision=dedup_gate against this event instead. |
| post_dedup | cogs/scheduled_poster.py | surface, guild_id, channel_id, reason, phase, post_preview — the MECHANICAL dedup gate (utils.dedup.duplicate_reason) discarded a unit that was already composed AND already through its self-gate. reason is the signal that tripped (same_link / text_similarity / shared_run / content_overlap), or source_seen (#3824: the double-post guard in ScheduledPoster._process_unit, answered by the surface’s _sched_source_delivered; music_news answers from its seen-set, and the music desk from its posted keys and, for a milestone, its stored rung (wave 15: a milestone’s ms: key was never stamped, so the slot and the fast pass both posted Rodeo’s 4B at 15:02:42 and 15:02:57 on 2026-10-17) – the slot and a watcher composed the same source post, and the second to deliver found it already delivered; a steady trickle is normal, a large count means the slot and the X watcher keep composing the same posts), or fact_claimed (wave 15, utils/chart_fact.py; music_desk, music_alert and, since wave 16, music_news): another surface already holds the claim on the same (chart, song, rank, ET date), or – desk and newsroom only – on the same streaming milestone (platform, track or album, song, rung; surface milestone_fact, no date), or on the same CATALOG count (platform, act, streams rung, song count; same surface, no date; desk catalog_stat and newsroom catalog milestones, since #4683), so this card did not send. Since #4683 the desk’s PLATFORM top-10 and streams boards (phase=platform_board) also claim the (chart, #1 song, 1, ET date) fact their headline states, so a board and an alert crown of one #1 no longer both post. Since #4505 the music alert also READS the claim (chart_fact.claimed) after its daily gate and BEFORE it composes, and spends one of the story’s 3 daily tries; that skip emits (at most 3 times per story per day) fact_claimed with detail.step = pre_compose (parse_json(detail)['step']) and costs no model call. Before it, the lane composed a claimed fact, the topic judge dropped it, and the next tick did the same: three Apple album debuts the desk had claimed were composed 19 times on 2026-10-23/24. Each row is one cross-surface repeat stopped. On music_news the phase is the story kind (chart / milestone), and the tweet is retired (music_news_retired reason=fact_claimed). Panel: desk_exact_guards; phase is the lane, so the loss is attributable. Why it exists: this gate ran with no event at all, so a composed-but-discarded unit cost a write call plus a judge call and left no trace. On 2026-08-04 the music desk composed 45 units and posted 5 in 140 minutes, and finding out why meant replaying real production lines through duplicate_reason by hand — 38 of 40 died here. Drop-rate = post_dedup events / units composed; a lane whose drop-rate approaches 1 is paying full price for posts nobody sees. Fires for EVERY ScheduledPoster surface, not only the topic-dedup opt-ins. A drop rate is not a material rate (#2037). A surface that does not CLOSE OUT the SOURCE of a deduped unit re-composes it every slot until the source ages out, so this event counts the loop, not the stories: the newsroom’s 892 composes over 7 days covered 95 distinct subjects, one written 136 times. Read post_dedup alongside dcount(matchup) on the surface’s *_scored event before concluding anything about volume — composes-per-distinct-subject is the number that says whether a surface is busy or stuck. _sched_on_dedup is where a surface closes the source out. The PUBLICATION lanes run a REDUCED signal set (#2098). chart_board / debut_sales / chart_proj / market_board are deduped on wording and links alone (_sched_dedup_wording_only), which drops shared_run and content_overlap (utils.dedup.FORMAT_SIGNALS) and the topic judge’s CATCH direction. Those two signals measure a post’s SHAPE, which on a recurring chart graphic is the FORMAT: a card names its chart, its week and its source every time, and “the Aug 15 Billboard 200, with HITS Daily Double” is a 40-char verbatim run before the post says anything of its own. Measured 2026-08-07 on the live Aug 15 week, the four projection boards all cleared the 0.6 gate and only ONE reached the room; after the opt-outs plus per-board compose steers, 4/4 did. So a post_dedup on one of those four phases should only ever read same_link or text_similarity — a shared_run or content_overlap there means the opt-out regressed, and is worth a look.|
| history_gate | utils/history_gate.py, taken by cogs/scheduled_poster.py + cogs/event_poster.py | surface, guild_id, ok, reason, duration_ms — the per-guild POST-HISTORY GATE (#2758), which serializes the history READ -> judge -> history WRITE walk so two wire desks cannot both read a history that holds neither’s post yet. Emitted on TWO branches only, so the steady state is silent: reason=waited (ok=true) is a real collision the gate serialized instead of double-posting, duration_ms being how long the second desk queued (since #3828 the gate is held per unit, so a wait is bounded by one unit, about 38s for a delivered card; a waited row near 150s again means a lane went back to holding it across a walk); reason=wait_timeout (ok=false) is the gate giving up after GATE_WAIT_SECONDS and posting on a possibly-stale read. THE GATE FAILS OPEN: a timeout is never worse than no gate, but it is the old race, so any wait_timeout row means a deliver is stuck for minutes and wants a look. An uncontended take emits nothing (~120 desk posts a day would otherwise be ~120 events of no signal). WHY IT EXISTS: on 2026-08-31 sports_desk and pop_desk both composed Messi’s international retirement on one tick, judged at 15:31:43 and 15:31:46, and tweeted at 15:31:48.389 and 15:31:52.396 — pop_desk read the history ~2s before sports_desk’s row existed. The dedup gate itself was healthy: it blocked the same story twice later that hour (15:37:29, 15:47:25), once the row was there. Panel: desk_history_gate. NO MONITOR: a waited row is the gate working (an alert on it would page on success), and wait_timeout is bounded by the deliver path, which post_deliver_failed and the latency ceilings already watch — revisit if wait_timeout is ever non-empty. |
| post_deliver_failed | cogs/scheduled_poster.py | surface, guild_id, channel_id, phase (the lane), post_preview — a gate-passed, non-deduped unit’s _sched_deliver returned False (#2063): a missing/unsendable channel or a failed send. This was the LAST silent branch of the ScheduledPoster spine — a dedup drop emits post_dedup, a delivered unit emits its surface’s *_posted event, and a failed deliver emitted nothing, so a gate-passed unit could vanish with no telemetry. With it the funnel closes as an identity: gate-passed = posted + post_dedup + this. That identity is how #2063’s ~890 “missing” newsroom units/month were diagnosed — they were dedup drops from before #2037 added the post_dedup emit, and the funnel has closed day-by-day since (measured 08-06..08-08: scored exactly = posted + deduped). Expected rate ~0. A sustained rate on one surface means its room config or send path is broken — the slot then re-fires next tick (nothing-delivered slots are not recorded), so a persistent failure also shows as repeated composes of the same subjects. |
| slot_muted | utils/schedule_calendar.py (note_muted), called by cogs/scheduled_poster.py, cogs/discourse.py and cogs/x_game.py | surface, guild_id, mood, reason=calendar_empty — the guild’s /menu calendar is MANAGED for this mood and holds no slot for this surface, so schedule_calendar.calendar_hours returns [] and _maybe_scheduled_post exits before any other event. Why this row exists: that exit used to emit nothing, so a surface muted ON PURPOSE and a surface that BROKE produced the same evidence — no events, no errors, no panel movement. It cost two diagnoses and a reverted PR. #3357 was filed as a p1 bug against sports_desk, pop_desk and cinema_news, dark 6 to 8 days, when the live post_schedule was {"yaps": {}, "chill": {}} — the owner’s deliberate mute of every surface not on a firehose set (steer 2026-09-19). #3491 read that same empty map as unmanaged, switched the desks back on, and was reverted the same night. How to read it: a surface with no events and NO slot_muted row today is one to investigate; a surface with a row is one a mod turned off. Rate-limited to a 6h HEARTBEAT per (surface, guild_id), not one row a day. The first version keyed the memo on the ET date, and a review caught why that fails: _dashboard_payload defaults every board to a qr-now-12h window, so the day’s single row ages out around midday and a continuously muted surface reads as broken again for half of every day — the exact ambiguity this row exists to remove. Six hours always leaves a row inside a 12h window and still costs 4 rows a day per muted surface against the ~1440 an unrated emit would write. ONE definition, three callers: ScheduledPoster, discourse and x_game all read this calendar and all had the same silent exit, so they call the shared helper rather than carry three copies. Two boundaries make it trustworthy: it fires on an empty resolved SCHEDULE, never on an empty due list, and only on [], never on None (no calendar for this mood, where the caller’s own default applies). Early in the day a real plan has nothing due yet, and firing there would emit for every configured surface every morning. Expected rate: one row a day for each surface a guild has muted — three a day at the time of writing. It does NOT change what the calendar does. Since 2026-10-15 surface=discourse stops: the retired discourse loop no longer runs, so it has no slot to mute (#1871). |
| slot_replan | cogs/scheduled_poster.py (plan_shrank) | surface, guild_id, channel_id, reason=plan_shrank, count (the stored posts_today), expected (slots due under the plan running NOW; folded into detail) — the daily-cap gate found today’s slot counter ABOVE the current plan’s elapsed-slot count and let the slot post anyway (#2921/#2926). Why the state is diagnostic on its own: the cap only lets a channel post while posts_today < expected, and expected only grows while the plan holds still, so posts_today > expected cannot happen under a stable plan. It means the plan SHRANK mid-day — a mod moved the surface to a lighter firehose cadence (scheduler_firehose_surfaces → scheduler_firehose_hourly_surfaces → scheduler_firehose_two_hourly_surfaces), changed the guild mood, or edited the /menu calendar. Before this escape the counter simply sat above the new elapsed count and the cap suppressed every remaining slot until the ET date rolled, returning BEFORE any log line and raising nothing: on 2026-09-03 music_desk posted 18 of its 30-min slots by 10:56 ET, a mod switched it to hourly 14 seconds after its last post, and the desk was dark for nine hours with zero error rows. surface_dark cannot see this class — that finding reads self-gate SCORES, and a surface stranded before compose produces none, so it drops out of the aggregation entirely rather than reading as a blackout. The escape is keyed on the most recent slot of the CURRENT plan, so it re-opens the gate once per new slot and the surface resumes the new cadence instead of draining the difference in a burst. Expected rate: a short burst on the day of a cadence change, then zero. A surface emitting it every day means its plan is being rewritten daily, or something other than the slot walk is bumping the counter. |
| topic_dedup | cogs/scheduled_poster.py | surface, guild_id, channel_id, reason, post_preview — the semantic TOPIC-dedup gate (a cheap Haiku judge, claude_client.topic_duplicate) SUPPRESSED a post as a RESTATEMENT of something already shipped. Opt-in via ScheduledPoster.TOPIC_DEDUP on the proactive news/market surfaces (sports_desk/pop_desk/cinema_desk/cinema_news/music_desk/music_news/market_drop); runs AFTER the mechanical duplicate_reason and catches the same STORY re-angled that order-sensitive token-overlap misses (a signing posted 3 ways scored 0.33-0.46 overlap; “72 Hours” across sibling Netflix boards 0.47 — both real prod cases). Policy (owner steer “allow real updates only”): a restatement carrying no materially-new info is suppressed; a genuine DEVELOPMENT (a new number/party/resolution) or a different subject is ALLOWED (the judgment tokens can’t make). It now ARBITRATES a mechanical hit as well as catching what the mechanical gate missed (#2037). It used to run only when duplicate_reason found nothing, so a mechanical hit was final — and that gate compares WORDING, which on a chart desk is mostly boilerplate: live it called a different artist’s identical-shape line a repeat, and the same song at a NEW chart position a repeat. An overturn emits dedup_overturned. The verdict is TRI-STATE because the two jobs want opposite failure directions — catching fails OPEN (never invent a duplicate), overturning fails CLOSED (never ship a repeat because Haiku was down) — so None means could not judge and both leave the mechanical verdict alone. The judge is handed the MATCHED post first, so an arbitration never depends on where in the list the match sits. Measured with the real judge on real history: 12/12 repeats confirmed, 3/3 cross-subject collisions overturned. HOW DEEP IT SEES (#2189). The judge keeps the newest claude_client.TOPIC_DEDUP_MAX_RECENT posts of whatever list its caller hands it. That ceiling was 12 and the wire desks read 80 rows over 36h, so the judge silently discarded 68 of them — and a truncated prompt still returns a confident verdict, so a clean “not a duplicate” and a verdict on a list that never contained the duplicate looked identical from telemetry. Measured on 2026-08-09: six desks post ~112/day into one shared history, so the newest 12 rows spanned 1h18m against the caller’s 15h19m. What it cost: three surfaces posted the same fact (“Dai Dai” by Shakira is #1 on Spotify’s Global Daily chart) at 17:19, 21:27 and 01:02 UTC on 2026-08-08/09, and in the 80-row list the third one read, the first two sat at depth 24 and depth 40. The ceiling now MATCHES wire_lanes.WIRE_DEDUP_READ_LIMIT (a test pins the two together), and a caller that exceeds it gets a WARNING log naming the drop. Raised 80 -> 180 (#2493) as the desk volume kept climbing: measured over the 14 days to 2026-08-23 the pool posts a median of 119/day (busiest 174), so 80 rows had shrunk to ~11-16h of span – under the same-day window this read exists for – and a 12-24h-apart cross-lane restatement fell past the cap (the newsroom posted the SAME Weeknd UK-leg wrap twice ~12h apart, the first at row 78 by the time the second composed). 180 rows restores a full 24h of span even on the busiest day and reaches 36h at the median; recall holds with no precision cost at the larger size (the buried repeat caught 5/5 at a 120/160/180-row read, live Haiku on the real pool). Re-measure and move it if the surface set grows again. A story that settles PERMANENTLY (a cert, a streaming milestone) still gets an exact key instead (music_news.cert_story_key / milestone_story_key) rather than a bigger window; the window is for same-day evolving numbers. Dilution was measured rather than assumed (scripts/ablate_dedup_depth.py, 5 runs a cell on that production history): the repeat is caught 0/5 at 12 and 5/5 at 24/40/80, while a candidate on a subject absent from the list ships 5/5 at every cap including 80. music_alert runs this judge too, off the same shared read (cogs/music_alert.py:_topic_repeat) — #2163 gave it the WRITE half of WIRE_DEDUP_SURFACES only, so it logged its posts for the desks and never read theirs; it emits topic_dedup (the semantic kind, not the mechanical post_dedup) with phase=the lane (on phase=settle also matchup and detail.ticker, #3889; a settle is judged only against the posts that state its settled figure, #3846), so it lands in the same desk_repeat_gate bucket as the desks’ judge verdicts and the suppress-rate math stays right. market_alert joined the same read on 2026-09-22 (#3638), for the same reason one surface later: its two pacing gates (_blocked, _forecast_paced) read only its own per-SERIES market_alert_state table, so music_alert posted the “Bass Persuades” pure-sales forecast at 02:48 UTC and market_alert posted the same Kalshi event at 04:34 UTC with neither able to see the other. It emits topic_dedup with surface=market_alert and phase=the trigger kind (swing / forecast_move / ending_soon / lead_change), so watch the desk_repeat_gate panel for a new slice rather than a new kind — no new event kind and no new field names, so the field budget is unchanged. Its REVEAL lanes (the RT scorecard, the ladder + entity outcome cards) write the shared history but do NOT run this judge, on the crown-lane reasoning: a result is new information by construction, and the judge would read it as a restatement of the forecast it settles. Emitted ONLY on suppression (its existence IS the suppression); the per-check count + latency ride claude_api purpose=topic_dedup, so suppress-rate = topic_dedup events / topic_dedup calls. FAIL-OPEN (a Haiku miss/error lets the post through, never swallows a real update). The mechanical content_overlap (#1681) stays the free deterministic first pass; this is the semantic backstop for the low-overlap re-angled cases it misses. |
| wire_collision | utils/wire_lanes.py collision_allowances (pop_desk, sports_desk, cinema_news; CB-29, 2026-10-18) | surface, ok, decision, reason, count — a wire post whose 5-token signature key was already posted, in OTHER words than every headline stored with that key (*_events.headline), went to claude_client.topic_duplicate. decision=allow (reason=allowed) = both of two calls said “not a duplicate”, so the story ranks although its key was posted; decision=drop with reason=signature_dup = a repeat, or reason=judge_error = a call returned None or raised (the old drop). count = the calls made (1 when the first said duplicate). ok=false = a call failed. One event per judged (key, headline) pair per process. The same words, or a key stored before CB-29 (no headline), drop with no call and no event. The calls ride claude_api purpose=topic_dedup, so they also count in that purpose’s call total; read the topic_dedup suppress rate with that in mind. Measured on 6229 real wire posts: 226 such pairs, 24 allowed (all distinct moments by hand), 196 dropped as repeats, 6 run-to-run flips. Panel wire_key_collisions (Desks board). NO monitor: every failure keeps the old drop. |
| discourse_skipped | cogs/discourse.py | guild_id, channel_id, reason (rate_limited/compose_error/empty) Retired 2026-10-15 (#1871): the discourse loop is dormant, so no new rows after that date. |
| discourse_icebreaker | cogs/discourse.py | guild_id, channel_id, category (ranking|hot_take|this_or_that|confession), score, reason, shipped, manual, trigger (fallback|quiet_room|manual), has_memory, has_topical, has_market, post_preview (a linkless opener; material-only, so a slot with no source material skips with reason=no_material; reason=room_unanswered = the human-activity floor held it back, fewer than MIN_HUMAN_MSGS_BETWEEN_ICEBREAKERS (3) human messages have landed since Toots’ own last post in the channel, so a second unanswered opener can’t stack on the first (enforced once in _icebreaker_fallback via a live channel-history read, shared by the scheduled + quiet-room paths, restart-proof; manual bypasses it); market line reused from compose or sports-room-gated. trigger names the source: fallback = an otherwise-empty scheduled discourse slot fell back to one; quiet_room = cogs.chimein broke a silent room (the inverse of a chime-in: fired off the chime-in tick when a listen channel went QUIET_AFTER with no human message, riding the chime-in mood/hours/cooldown/cap gates, one opener per lull); manual = on-demand /discourse category:icebreaker (also manual=True). icebreaker is a permanent surface (no per-guild experiment): a ship-worthy opener (clears the quality floor) posts straight to the room — shipped=True; a gated/no-material slot is recorded by this event with shipped=False.) |
| chimein_evaluated | cogs/chimein.py | guild_id, channel_id, decision (mood_off_gate/hours_gate/vibe_gate/cooldown_gate/daily_cap_gate/threshold_gate/reacted/empty_generation/dedup_gate/quality_gate), vibe, score, mood; situational: signal (same_link/text_similarity/shared_run/content_overlap, on dedup_gate), local_hour_et, quality_score, quality_reason, post_preview, voice_present (a voice note in the buffer lowered the worth-posting threshold — likelihood only, the quality gate is unchanged) |
| chimein_posted | cogs/chimein.py | guild_id, channel_id, score, vibe, hook, mood, quality_score, quality_reason, voiced (shipped as a native voice note), sing (sung delivery), post_preview (the shipped take, POST_PREVIEW_CHARS — graded by the live-log eval pass), voice_present (a voice note in the buffer lowered the post threshold) |
| quiet_room_decided | cogs/chimein.py | guild_id, channel_id, decision (break|hold), score, reason (the verdict on cold-opening a quiet room — the inverse of a chime-in. After the cheap silence/mood/hours/cooldown gates, an early deterministic human-activity pre-gate runs first — reason=room_unanswered (score=0.0, no model call) holds when she posted recently and fewer than MIN_HUMAN_MSGS_BETWEEN_ICEBREAKERS humans have spoken since, so the path skips the paid quiet_room_score/produce_icebreaker work; otherwise a Haiku quiet_room_score call (gated on the same mood-tuned threshold as chime-in, 0.7 chill / 0.5 yaps) decides whether the room is genuinely dead and worth reviving, weighing whether Toots was just mentioned/engaged. break = a quiet-room icebreaker is being produced (→ discourse_icebreaker with trigger=quiet_room, called with skip_activity_floor=True so the shared floor isn’t re-read), hold = left alone. Bias is toward hold; emitted once per lull) |
| link_enrich | utils/link_enrich.py | platform, url_host, ok, duration_ms, cache_hit |
| link_check | utils/link_enrich.py | platform, ok, alive (False only on confirmed 404/410), status, duration_ms — the dead-link HEAD guardrail’s per-URL liveness probe (distinct from link_enrich’s metadata fetch) |
| github_api | utils/github.py | op (create_issue|get_issue|list_issues|close_issue|comment), ok, status, duration_ms — one GitHub REST call (the /order path + the feedback log) |
| feedback | cogs/order.py | guild_id, user_id (the regular), logged; on logged=True: kind (request|suggestion|complaint|praise), issue_number; on logged=False: reason (daily_cap|github_error) — Toots logged SOFT feedback via the /ask note_feedback tool, recorded as a comment on one rolling feedback-log issue (label feedback, plain body so claude-code-action ignores it) for mods to triage. The lighter lane beside file_fix: no preflight/kitchen/pipeline gates, just a per-guild FEEDBACK_DAILY_CAP |
| fix_update | cogs/order.py | guild_id, user_id (the regular), logged; on logged=True: issue_number, order_id; on logged=False: reason (unknown_order|terminal|daily_cap|github_error) — Toots appended NEW INFO from a regular’s follow-up reply onto an existing in-flight fix-order’s issue via the /ask update_fix tool (add_fix_update), as a plain attributed comment (no @claude, so it records context without spawning a duplicate PR run). The counterpart to file_fix (file opens the order, update adds detail); scoped to this guild’s non-terminal orders, runaway-guarded by the per-guild AUTO_ORDER_DAILY_CAP (shared with file_fix, separate counter) |
| railway_api | utils/railway.py | op (list_deployments|redeploy|deployment_logs|build_logs|gql), ok, duration_ms — one Railway GraphQL call (the /undo rollback path AND the read-only check_deploys /ask diagnostics tool) |
| tool_call | cogs/ask.py + cogs/message_search.py + cogs/discord_info.py + cogs/playercard.py | tool (the /ask tool that ran: break_down_board, lookup_player_props, scoreboard, api_sports, standings, matchup_preview, box_score, game_plays (#3404: a named game’s scoring plays + last plays; matched = resolved on the slate, has_events = a scoring log rendered, count = last plays rendered), game_markets, match_highlights, list_sports, game_lookup, engagement, discord_lookup, look_at_image, find_image, find_media, listen_to_audio, read_media, search_messages, self_serve, check_deploys, music_forecast — the market’s tracking-week projection for one release off the desk’s Luminate ladders, #3392: matched = readings returned, ok=false with matched=0 = no live ladder or a thin book; chart_history — the stored past for an act or one release off the chart-history store, #3562 E: matched = store rows read (0 = nothing stored for the act, still ok=true), ok=false = the DB read raised; source=x_mentions when the X mention reply lane made the same read for its facts block, with count = the history lines it handed the draft; detail.outlook (#3562 B) says whether the block carried the store’s own next-week projection, which is the one signal that catches the silent case – blocks still rendering while every projection is gated off by a stalled walk, a store with no printed units, or a curve cell that fell under its pair floor. The dashboard’s tool_outlook panel graphs it), ok, + the promoted shared dims where they apply (query, matched, count, reason, phase, sport, aspect, action, source, guild_id, user_id, host, url_host, error) + a demap’d detail dict for tool-specific fields — the UNIFIED on-demand /ask tool-run event (#1296: folded ~20 near-identical per-tool *_lookup kinds into one parameterized event keyed by tool, via utils.events.emit_tool; the tool-volume/hit-rate dashboards group by tool, tool-specific signals ride detail off the field budget). memory_search (recall dashboard), reference_lookup + catalog_lookup (integration-health), and social_search (timed latency) deliberately stay their OWN kinds |
| auto_order | cogs/order.py | guild_id, user_id (the reporting regular), filed; on filed=True: issue_number, order_id; on filed=False: reason (gated = kitchen-closed/pipeline-red/in-flight cap/AUTO_ORDER_DAILY_CAP daily budget; plumbing/refused = preflight rejected; preflight_error/github_error = downstream failure) — Toots filed (or declined to file) a fix-order ON HER OWN via the file_fix /ask tool off a regular’s bug report. The autonomous counterpart to a mod’s /order; does NOT consume the server order budget (its own tighter daily cap) and still honors kitchen/pipeline/in-flight gates + the order preflight (protected paths) |
| health | cogs/health.py | integration (the watched source: apple_music, the market/odds feeds api_sports/sgo/the_odds_api/polymarket/kalshi, plus the #968 app-backbone set embeddings/elevenlabs_tts/elevenlabs_stt/perplexity/twitterio/odds_divergence), action (filed|dedup|alert|usage_alert); crater actions also carry failure_rate + attempts (the numbers it judged on) + recent (#725: True = a sharp RECENT-tail crater the full-window average still hid — assess checks the last _RECENT_TAIL_EVENTS calls too, so a fresh throttle self-announces while it’s fresh instead of waiting ~15 min to dominate the window; this is why she stayed quiet on the SGO blip before). action=filed: a fix-order filing was attempted — ok=True with guild_id + issue_number + order_id on success, ok=False with reason (gated = every guild blocked by kitchen/pipeline/cap; plumbing|refused|preflight_error|github_error = the order text itself rejected) on a miss (the filing also emits auto_order). action=dedup: a crater was found but already filed within the 24h window, held back. action=alert: an owner-facing crater alert posted to #bot-logs (guild_id) with an @master ping, delivered=bot_logs (#968: #bot-logs ONLY, never a content room), gated by master switch + a per-(guild, integration) durable cooldown — deliberately NOT mood-gated (an ops alert isn’t proactive content). Fires for every crater, incl. the ops-only (notice="") backend watches that used to file a fix-order silently — the self-healing watch (#443) now also reaching the owner, closing the gap the drained twitterapi.io wallet exposed. delivered=suppressed (reason=backstop_healthy): the crater has a backstop integration (a Watch.backstop) that’s provisioned + healthy and carrying the load, so the owner alert is held (the fix-order still files for ops) — it only fires when the backstop is ALSO down. The SGO betting-lines alert rides this (#748): once the Odds API SGO-down backstop shipped, SGO going flaky no longer drops betting lines, so she stays quiet unless the_odds_api is degraded too (“everything flaky and down”). action=usage_alert: a metered-wallet early warning posted to #bot-logs (guild_id, integration=the source) with an @master ping — a prepaid wallet under its floor (twitterio/scrapecreators) or a capped source ≥85% of its limit, caught before the 402/429 wall (deduped per guild: a prepaid wallet on usage:<source> with a 12h window; a capped source on usage:<source>:<period>:<level>, level low|full, with a 7-day window for a month cap and 12h otherwise, #4186) |
| db_query | db.py (_run) | op (coarse sql_op() label, never params), method (execute|fetch|fetchrow|fetchval), ok, duration_ms — one DB query through the central choke point (singleton queries; transactional paths bypass _run). SAMPLED: only emits for queries ≥ _DB_QUERY_EMIT_FLOOR_MS (50ms) or any failure, so the hot-path sub-ms majority doesn’t flood logs (method=transaction / op="TXN <table>" rows emit every call). So these events are a SLOW-CALL SAMPLE with no denominator (#2257): the fast calls are not counted, and a percentile over the events is a percentile of slow calls only – one 1s call among thousands of 2ms calls read as a 1s “p99”. The sample gives two exact values, and every consumer uses only those: the COUNT of calls at or over a threshold of 50ms or more, and the MAX. The ops monitor keeps db_query OUT of the p50/p95/p99 latency map; it keys a separate db_slow table by op (sampled, fails, calls ≥ DB_SLOW_CALL_MS 1s, max) and files a db_slow finding (an optimization candidate) when a shape has ≥ DB_SLOW_CALLS_MIN (5) calls at or over 1s in the window (measured 2026-10-25 over 12h: 8,770 sampled calls; SELECT FROM kv_cache 161 over 1s, four shapes 10-20, seven shapes 1-4). The Axiom Database board shows the same counts + max, and its two monitors count 2s+ calls (> 10 in 30 min) and 5s+ calls per shape (≥ 3 in 1h) instead of a p99 |
| span | utils/instrument.py (@timed / timed_span) | name (function / flow-step label), ok, duration_ms — the catch-all latency primitive for heavy functions (feeds.recent_messages, markets.get_context, the memory write pipeline) and discrete flow steps (ask.parallel_fetch, ask.memory_context, ask.vector_recall, recap.produce_recap, …). Ops monitor keys these span:<name>; report-only (no per-span ceiling), but the trend gate catches a climbing tail |
| pplx_<purpose> (pplx_ask/pplx_discourse/pplx_recap/pplx_chimein/pplx_music/pplx_icebreaker/pplx_trending/pplx_cinema_desk) | utils/perplexity.py | ok, duration_ms; on success: input_tokens, output_tokens, response_chars, hedged, source_count, context_size, recency; on failure: error. pplx_icebreaker is the idea-tuned conversation-starter hook fetched per quiet-room opener (surface="icebreaker": current debates/divisive/nostalgic angles to springboard off, never cited, always fresh, fail-open). pplx_trending (#1272) discovers the SPECIFIC current viral moments that fit a channel’s vibe (surface="trending", this-week), distilled by parse_trending_phrases into short searchable phrases the trending-clip gather searches — the fix for the raw-topic-string clip query |
| grok_search | utils/grok_search.py (#1390) | purpose, model, ok, duration_ms; on success: input_tokens, output_tokens, count (x_search calls), + folded detail: response_chars, source_count, cost_usd, x_posts_fetched, x_users_fetched, cached_tokens, reasoning_tokens; on failure: error, http_status. detail.cost_usd is xAI’s own billed cost of the call (#4720), read from usage.cost_in_usd_ticks. It includes the per-post ($5 / 1,000) and per-user ($10 / 1,000) fetch fees, which were 90% of the 2026-10 bill and which no token count shows. It is absent (not 0) when xAI does not return it. Read spend with summarize sum(todouble(parse_json(detail)['cost_usd'])) by purpose. The ops monitor’s X-search (xai) line sums it by purpose, the LLM board’s spend panels include it as vendor xai, and the monitor Grok X-search spend over $75 in 24h fires at about twice a normal day. Spend from a script run outside Railway does not reach Axiom, so the xAI console stays the full total |
| hedge_check | utils/perplexity.py hedged_checked (CB-22, 2026-10-18); callers music_desk._perplexity_context, music_news._pull_artist_news, music_news._perplexity_research | purpose (music_desk | music_news | music_news_verify), ok, decision, reason. A Perplexity answer that hit the _HEDGE_MARKERS phrase list went to Haiku perplexity_found_answer (claude_api purpose=pplx_found_answer). decision=keep (reason=found) = the answer states facts, so the lane keeps it (the list alone dropped it before); decision=drop with reason=none (a real non-answer) or reason=no_verdict (an error, ok=false; the list’s drop stands). An answer with no list hit makes no call and no event. Measured on 35 real desk reads: the list hit 5, all 5 stated facts, and Haiku kept all 5 in 2 of 2 runs; 3 non-answer goldens dropped. About 5 a day (119 desk + 37 newsroom list hits in the 30 days to 2026-10-18). Panel hedge_check. NO monitor: a failed call keeps the old drop. |
| link_stripped | claude_client.py (discourse, ask, recap, music_post, chimein_post, discourse_icebreaker, commentate) | purpose, reason (hallucinated | redundant | dead_link), count, urls (openers + live commentary are linkless, so any URL is stripped as hallucinated) |
| discourse_relabel | claude_client.py (react_to_trending) | purpose (discourse_trending), reason (scaffolding_leak | unresolved_pick), ok — a trending clip react echoed its internal candidate label into the shipped text (“clip B out here selling…”, #clip3-leak; the [A]/[B] tag that binds a vision frame to a list entry, meaningless to a reader). The PICK was fine, only the wording leaked, so instead of discarding a good clip it re-reacts to JUST the picked clip (resolved by the link the reaction ended on) — a single clip carries no letter scaffolding, so the retry structurally can’t leak. ok=True = the clean single-clip retry shipped; ok=False = dropped (the slot falls through to a normal discourse take): reason=scaffolding_leak when the retry itself came back empty, reason=unresolved_pick when the leaked reaction carried no clip link to pin the pick to. Bounded to one extra call (_allow_retry), never a loop |
| market_fetch | utils/markets.py + utils/api_sports.py + utils/the_odds_api.py + utils/espn.py | source (sgo/polymarket/kalshi/api_sports/the_odds_api/espn), query (volume_batch:N = the move lanes’ trade read, #4714 KalshiClient.get_recent_volume, which fails OPEN, so the ops monitor gives it its own kalshi_trade_read health rate apart from the per-source line), ok, duration_ms, cache_hit, result_count, error (for the_odds_api a call skipped by the client’s own gate is no_key or, since #3552, quota_dead – a 401 put the client to sleep for 6h), http_status (every sgo AND kalshi fetch miss classifies error off the HTTP code via _classify_fetch_error — rate_limited 429, forbidden 403, server_error 5xx, not_found 404, unreachable no response/a breaker short-circuit — instead of a bare fetch_failed, order #722/#73 for sgo, order #77 for kalshi: a 429 quota throttle, a 403 datacenter-IP block, and a plain timeout used to be indistinguishable in the same bucket). source=kalshi query=open_events is the hourly catalog walk; its error is
pagination_failed (a page gave up after retries) or, since #3553, truncated
(the walk hit _KALSHI_OPEN_EVENTS_MAX_PAGES with a cursor still outstanding, so
the catalog has outgrown the cap). Either way the walk returns nothing and the
prior FTS table is KEPT; a truncated row means discovery is going stale and the
cap needs raising. Before #3553 a cap hit was silent and the partial set was
rebuilt over the good one, so events past the cap simply vanished from discovery.
source=espn carries http_status on a failure since #3584 (a 429 throttle vs an outage vs a bad path — previously every miss was a bare fetch_failed), and its breaker cooldown escalates per failed probe, so a run of circuit_breaker integration=espn transitions should now SPREAD OUT rather than repeat on a fixed 60s beat; a flat 60s cadence means the escalation is not being reached. source=espn (#1113): the FREE, keyless ESPN scoreboard, the last-resort score/finals backstop for the SGO-only individual sports — tennis (only when SGO degraded) + MMA/UFC (only when API-Sports degraded); query=sport (tennis|mma), result_count=matches parsed, error=all_feeds_failed. Reuses market_fetch so the ops-monitor’s per-source integration health + the ops-only espn health Watch cover it with no new wiring. Team-sport fallback (#full-espn-fallback): EspnScoreProvider now ALSO backstops the TEAM sports (soccer / NBA / WNBA / MLB / NHL / NFL) — live scores + just-finished FINALS off ESPN’s day scoreboard (source=espn_results, one market_fetch per league/day) — gated on api_sports.degraded, so an API-Sports outage (a suspended account, a crater) no longer darkens live commentary or bet settlement for those sports. The finals carry REAL scores, so the Bookie settles off them through the shared idempotent _settle_game (a level score PUSHES, never a wrong payout). Bookie pregame-odds fallback (#espn-odds-fallback): the SAME espn_results fetch now ALSO feeds the Bookie’s LAST-resort price fill (_supplement_with_espn_odds) — for an UPCOMING game still unpriced after SGO + the Odds API + the prediction markets, ESPN’s FREE scoreboard moneyline is folded on (odds_source='espn', American values), so the game is bettable instead of hidden. Fill-only-when-absent (a fresher SGO/Odds line always wins), pregame-only (ESPN has no in-play line, so the stale-line router reprices it off predictions if it goes live), and fold-only (it never INTRODUCES a game no settleable source surfaced, so it can never strand a bet). No new event — the fold’s ESPN reads are the existing espn_results market_fetch, so the ops-monitor integration health already covers it; the odds_source='espn' stamp on the bet is the per-bet audit trail. Verified vs the Odds API consensus on 15/15 live MLB games: favorite agreement 100%, implied-probability delta within ±1.5%. source=espn_summary (#espn-depth-parity): one game’s summary?event= payload – the boxscore + key events + rosters + season series that backstop the commentator’s per-game DEPTH (goal/card EVENTS, team STATS like possession/shots, and PREGAME formations + lineups + head-to-head) when API-Sports is degraded. EspnScoreProvider.game_events/game_team_stats/game_pregame parse it (one fetch shared per tick, cached per event); game_standings reads ESPN’s standings endpoint, and game_leaders reads the summary’s pre-computed leaders (basketball’s “who’s cooking”; soccer/baseball have no top-level leaders and stay empty rather than index-guess a stat off the raw box score). All display-only – the commentator degrades to the scoreline on a miss, never blocks a post. source=sgo reason=budget_shed (#786): the proactive budget governor (SportsGameOddsClient._shed_heavy_call) DROPPED a heavy league-wide read — the full game-board or league-wide player-props (the priciest calls on the tight rookie-tier monthly entity cap) — because the entity budget is near the cap; NO network call (cache_hit=True → excluded from integration-health like a no-op guard), result_count=0, ok=True, caller degrades fail-open (no board/props). The proactive complement to #1177’s scoreboard-cadence throttle: #1177 slows the CORE fetch, this drops the OPTIONAL depth first, reserving budget for the cheapest critical reads. source=sgo reason=quota_paced (#3229): the month-pace governor (SportsGameOddsClient.over_month_pace / _paced_out) SKIPPED an SGO read at ANY entry point (event odds, props, board, the in-season probe, the leagues catalog) because the month’s month_entities usage is ahead of an even spread through the UTC calendar month, or the cap is spent. Same shape as budget_shed: NO network call, cache_hit=True (excluded from integration-health), ok=True, result_count=0; a live-score read still serves a slate younger than the stale bound. While paced, sgo.degraded reads True, so every consumer takes its existing ESPN / The Odds API fallback. Why it exists: SGO moved from the paid rookie tier (100,000 entities a month) to the free amateur tier (2,500) on 2026-09-11, and the fan-out then spent each month’s 2,500 in about one day and 429’d for the rest (about 144 real failed requests and 700 failed market_fetch rows a day in October). The quota itself stays visible on the quota event (metric=month_entities) and the ops-monitor near-cap finding. source=api_sports errors (#still-no-art): API-Sports answers an ACCOUNT / PLAN / RATE problem with HTTP 200 + a populated errors object (a suspended account → {"access":"…suspended…"}, a plan gate → {"plan":…}, a throttle → {"rateLimit":…}) and an EMPTY response. _fetch used to hand that back as a healthy [], so a suspended account read as ok=True “nothing live” across every api_sports call (games, live scores, team logos) for days while sports GAME cards shipped art-less — the ops-monitor never saw it. _response_error now turns a populated errors into ok=False (the breaker opens + fallbacks take over), so the outage shows in the api_sports integration-health rate. source=espn_logos (#still-no-art): the KEYLESS ESPN team-crest backstop for the market card’s sports art – when api_sports.team_logo returns nothing (a down / suspended account), EspnClient.team_logos reads the crest off ESPN’s /teams endpoint (query={home} vs {away}, sport, result_count=2 on a matched pair else 0). Both sides must resolve inside ONE of the sport’s ESPN leagues or it returns None (the card keeps the branded floor, never a wrong crest); the team list is cached per league path (static), so this costs a fetch only on the first miss per league. Single-team crest (#full-espn-fallback): EspnClient.team_logo(name, sport) is the one-team analog — the deterministic floor-beater for a card whose subject is ONE team (a futures winner like “MLS Cup Winner” -> its leading club, a standings/award market), and the ESPN fallback inside the wire-desk team card (entity_image._team_fallback_card). It sits BELOW the vision photo search in the market ladder, so a real team PHOTO wins first and the crest only fills a card otherwise heading for the branded floor; exact-name-matched inside the sport’s leagues, so never a guessed crest |
| billboard_fetch | utils/billboard.py | source (billboard), query (<chart>:<week\|current>), ok, duration_ms (the READ and its retries only: the limiter token is taken BEFORE the event opens, the reference_fetch rule. Before #4368 the queue was inside it: the music desk’s source watcher reads 8 charts at once behind ONE 6/min limiter, so the reads landed on exact 10s steps and the last one reported ~60s. Measured 2026-10-17 on the real 8-chart read: max 60,084 ms before, 664 ms after, same 60s wall time. The same applies to billboard_artist_history and billboard_story), count (rows parsed), cache_hit (two-speed durable cache: ~12h when current, ~20min while a flip is due); on a miss: reason + error + http_status — one FIRST-PARTY Billboard chart read (Hot 100 / Billboard 200 / Global 200 / Artist 100 / …, one parser for all). reason: blocked (403/451 — billboard.com REFUSING us; its robots.txt names anthropic-ai, an EXPECTED end-state — docs/BILLBOARD_SOURCES.md §5), shape_change (a 200 that parsed to near-nothing = a redesign, the SILENT failure), partial_credits (a 200 whose song/album rows carry an EMPTY artist — the page read MID-PUBLISH: Billboard lands the rows first and attaches the artist links on new entries minutes later; measured 2026-09-09, the Sep 12 Hot 100 one minute after its flip carried 15 of 16 debuts with no credit, was cached 12h, and posted as “Take Care leads 16 new entries … from 2 different acts”. Refused like a shape change: nothing served or cached, a cached week with a credit-less row is re-read too, the next poll re-reads, the block clears itself on the full page; dropped in detail = rows with no credit. Artist-unit charts are exempt — their artist cell is empty by design), http_error, breaker_open, unpublished (#4769: a DATED read of a week Billboard has not printed yet. A week after expected_chart_date is not read at all, and the due week read before its Tuesday flip comes back as a 200 with no rows and no week line. Billboard used to answer that date with the current chart; since 2026-10 it answers with that empty page, and it read as shape_change: 1,905 reads of billboard200:2026-10-24 over 2026-10-18..20. Not a block, no error, and the ops monitor leaves it out of the rate and the per-chart run). Panel bb_fetch splits the reads current vs dated by reason. Each dated page keeps its own failure streak (keyed by the query, <chart>:<week>, in BlockState.streaks; follow-up on #4769): with one key per chart a success on any other page of that chart reset the streak, so latin:2026-10-17, empty on every poll on 2026-10-25, logged consecutive=1 and an error 20 times that day. billboard.com serves the empty page for PUBLISHED dated pages too, for hours, through its CDN (measured 2026-10-25 08:00Z: 16 of 24 dated pages across 8 charts empty, a cache-busting request empty too, every current page full). So a dated week that read cleanly is cached BILLBOARD_CACHE_TTL_DATED_DAYS (14) days, not 12 hours: a published week cannot change. A page whose weeks caption was not read (2026-10-27: Billboard renamed “WEEKS” to “WEEKS ON CHART” and every row read weeks_on=1) is served, not refused, and emits error with source=billboard_weeks, recoverable=true on every such read (billboard.weeks_unread: every row with a last-week position reads 1 week). There is deliberately NO stand-in source (owner call 2026-08-04: the GitHub-mirror fallback was removed — it was itself a once-daily crawler of the same site, so it added a hop and staleness, never data): a block yields nothing, loudly. blocked + shape_change + partial_credits also emit error on the first failure of a streak and every 10th after (recoverable=False for the CURRENT page; recoverable=True for a DATED page since #4769, because the current page still serves the live lanes and only the week-over-week comparison drops out), and the ops monitor raises a source_blocked finding naming the action (high for a refusal / a sustained or still-dark run; medium for a partial or empty page that completed within the window) |
| billboard_artist_history | utils/billboard.py | source (billboard), query (<slug>:<code>), ok, duration_ms, count (entries parsed = the artist’s entry count on that chart), cache_hit (~6h durable, its own namespace); on a miss: reason + error + http_status — one per-artist CHART-HISTORY read (/artist/<slug>/chart-history/<code>/), the CAREER-ENTRY count a weekly chart page cannot carry (#2154 follow-up). Only the two flagship charts are mapped (billboard200→tlp, hot100→hsi). reason: blocked (403/451, the same §5 refusal as billboard_fetch), unknown_slug (404 from a GUESSED slug — an ORDINARY fail-open miss, NOT a Billboard block), no_entries (a 200 that parsed to nothing — a soft-404 or a redesign), http_error, breaker_open. detail.slug_source (2026-09-15, #3331) says WHICH page was asked for: chart_row (the slug Billboard printed in the act’s own chart row, read out of the cached week) or guess (artist_slug, the fold of the credit string). The guess does not only 404 — it lands on ANOTHER ACT: BLACKPINK’s LISA is filed as lisa-of-blackpink, so “lisa” answered 200 with a 1980s act’s three Hot 100 entries from 1982-84, and the debut guard read that page as hers. Watch the guess share: it is the fraction of these reads that can still be another artist’s, and it rises when acts reach the read without a row on any chart week we hold (the slug is only remembered for the current week). Deliberately QUIET, unlike billboard_fetch: a miss suppresses one optional “their Nth entry” clause, never a post, so it does NOT touch the weekly-chart block_state() and does NOT emit error — a wrong-slug 404 is not Billboard refusing us. Watch the ok-rate as the source-health signal; a sustained blocked here (not unknown_slug) is the AI-agent block reaching the artist pages too |
| billboard_story | utils/billboard.py | source (billboard), query (story_posts:<week> = the WordPress post-API discovery read, story:<week> = the story page), ok, duration_ms, count (posts in the window | 1 for a page); on a miss: reason + error + http_status — one Billboard chart-STORY read, the chart-history store’s source for the printed units of the Billboard 200 top 10 (research round 2, 2026-09-20; utils/billboard_stories.py parses the prose, cogs/chart_history.py’s story walk verifies every figure against the stored page rows before writing it). Two reads per chart week, ever: the Chart Beat posts in the week’s publish window (/wp-json/wp/v2/posts, category 8546590, the Sunday..Thursday before the chart Saturday), then the one story page. reason: blocked (401/403/451, the same §5 refusal as billboard_fetch), shape_change (the post API answered something that is not a post list), http_error, breaker_open (its OWN breaker: a failing story endpoint never opens the chart breaker). Deliberately QUIET, like billboard_artist_history: a miss leaves one week’s units unfilled and the walk retries it next tick, no error and no block_state(); the ok-rate (billboard_story in the misc integrations) is the health signal, and a sustained blocked here is the AI-agent block reaching the story pages. Panel bb_story |
| music_news_debut_framing | cogs/music_news.py | ok (the debut was CONFIRMED), reason (presence | history | unconfirmed), guild_id, phase (chart), matchup (the subject), source (the wire account), via (how the wire’s debut call was read: classifier | bounded | regex, #3418), + folded detail: chart (the chart the history was read on), entries (rows that page returned), event (the resolved chart event: debut / reentry / move / hold / “”) — did anything CONFIRM that a chart story is a NEW ENTRY (owner steer 2026-09-01). The gap it closes: chart_presence settles debut-vs-established for the live PLATFORM charts only, so a weekly Billboard chart came back is_debut=null and the wire’s word rode through as Toots’ own claim. On 2026-09-01 both “Painted You Pretty debuts at No. 93” and “All My Exes … debuts at No. 92” shipped off @billboardcharts while our own Hot 100 read was waiting_for_flip — the site still served the 08-29 chart, so music_news_chart_lookup said absent and nothing had checked either claim. reason=history is the new confirmation: Billboard’s own artist chart-history page (billboard_artist_history) carries a debut DATE, and it updates EARLIER than the chart page — measured 2026-09-01, it already listed “All My Exes … 9/05/26 … 92”. The date is compared against the CURRENT chart week (billboard.expected_chart_date, the one definition of that), never a rolling window of days: Billboard dates a chart to a Saturday and reveals it the Tuesday four days before, so on reveal day LAST week’s chart date is only three days past and any window wide enough for this week would confirm a second-week title as a debut (Codex review). detail.rewrote says the story RECORD was rewritten too — news_compose_blob builds it from the claim and the self-gate grades the take against it, so a record still reading “debuts at No. 93” is an answer key that marks the debut correct; the claim becomes the bare position through the same settled_claim formatter a live reading uses. reason=unconfirmed hands the compose unconfirmed_debut_note and the take states the bare POSITION in the owner’s wording (“is at No. 93 on the Billboard Hot 100” – the note SHOWS that sentence, naming the position the story reports, because describing the wording got “sits at” instead): the wire’s number still ships, only the unverified claim about the title’s chart HISTORY is dropped — prefer absent over invented. It also vetoes the career-ENTRY ordinal (music_news_entry_ordinal claim=entry), so a post that may not say “debuts” cannot say “their fourth entry” either. Filed only when the wire framed the story as a debut or we confirmed one, so an ordinary climb story never files a row. WATCH the history share of the confirmed rows: it is the ONLY way a Hot 100 or Billboard 200 debut keeps its wording, and it rides a GUESSED artist slug — an act Billboard slugs differently 404s (Hudson Westbrook did) and reports flat. A history rate falling toward zero means the slug guess or the page parse broke, and every real weekly debut is being under-reported |
| music_news_entry_ordinal | cogs/music_news.py | ok, phase (chart), matchup (subject), count (the artist’s entry count), + folded detail: chart (the CHART_SOURCES key), claim (peak | streak | weeks | entry) — the newsroom stamped a chart take with a grounded CAREER fact read off the chart’s own pages (billboard_artist_history for peak / entry, the weekly Billboard row for weeks, the kworb daily row’s (x?) days-at-peak column for streak), instead of a guess. detail.claim=entry (#2154) is the debut ORDINAL (“JUNGLE’s fourth Billboard 200 entry” instead of the guessed “first”); it fires only for a debut music_news_debut_framing CONFIRMED (the count is the title’s ordinal only when it is the newest entry, and a new-entry claim she may not state in words must not ship as an ordinal either) and its grounding converts a career_ordinal DROP into a ship. detail.claim=peak (#2587) is a career-BEST milestone (“their first #1”, “their highest-charting entry yet”) off the peak column + the CURRENT position, so it is fresh news and can fire on a CLIMB, not only a debut; it takes precedence over the entry ordinal (one clause ships). It fires only the week the position is REACHED (owner steer 2026-09-08): the settled row’s last_week / weeks_on ride ChartPosition, and a held position, a fall, or a re-entry yields no peak claim — “Choosin’ Texas … her first #1” had shipped off a row reading last week #1, 45 weeks on. detail.claim=weeks is the rung a HELD position gets instead (music_news.chart_run_fact, “in its 45th week on the Billboard Hot 100”, off the row’s own weeks column, a digit ordinal so the digit gate grounds it). detail.claim=streak is the daily twin (owner example 2026-09-08): “has 11 days at #1 on the global Spotify daily chart in total” for a held Spotify #1 that is its own peak, off music_news.chart_streak_fact (the day’s plays ride the settled reading itself: “#1 on the global Spotify daily chart with 5.4M streams on the day”). A held daily #1 reaches the compose only when its streak CROSSED a milestone bucket since the last report (music_news_filtered reason=chart_repost now folds detail.streak, the days-at-#1 count the gate saw; MUSIC_NEWS_STREAK_MILESTONE_DAYS, default 7), so on spotify_global / spotify_us expect a streak row about once a week per long-running #1 and chart_repost rows in between. music_news_filtered reason=streak_unstated is the streak-triggered repost whose composed line did NOT state the streak: dropped and marked seen rather than shipped as a bare repeat of the held #1, and the milestone is not consumed (detail.chart, detail.streak). Measured 0/25 in the dry runs (the compose stated the clause every time), so a rising rate here means the compose stopped folding the CHART HISTORY clause in. Watch the peak : weeks split on hot100: a peak row with the same subject in consecutive weeks is the stale-milestone bug back; weeks should be the common kind for a #1 story, peak the rare one. Both history kinds require the reported title on the page (the wrong-artist guard, music_news.entry_ordinal_fact / peak_history_fact), emit at SHIP time (after the gates + self-score), and only when the composed line ACTUALLY states the fact (music_news.career_clause_in_line) — so the rate is how often a grounded career fact shipped. Watch claim=entry against music_news_scored reason=career_ordinal (the drop): a rising entry rate with a falling drop rate is the fix working. Watch claim=peak on its own — a #1 / career-best is rarer and higher-value |
| feed_fetch | utils/industry_feeds.py | source (the feed key: pollstar|iq|mbw|billboard_news), ok, duration_ms, count (items parsed), cache_hit (~15min durable); on a miss: reason (unreachable = no 200 | unparseable = a 200 with zero items, i.e. a feed shape change) + error — one trade-press RSS read (#reporting-oversight). The FIRST-PARTY news source, read directly so a number is in hand rather than chased through a web-confirm that lags a fast story (the R&B Tour gross case: the wire said $153.4M/20 shows, the web-confirm returned the stale $101.9M/13 shows, and Toots posted the stale one). One parser for all four feeds, namespace-driven. Each source has its OWN breaker (integration=industry_feeds, source=<key>) + limiter so one host refusing us does not blind the others. Fail-open (a miss → the desk keeps its X wire) with emit_error(recoverable=True); DARK until a desk consumes it. Per-source ok-rate is the health signal a single feed went quiet |
| feed_article_image | utils/industry_feeds.py | source (the feed key), ok, duration_ms; on a miss: reason (unreachable | no_og_image = the page no longer names an og:image, a shape change) — one read of a feed story’s ARTICLE page for its lead photo (2026-09-30, the Rauw Alejandro “Macacoa 2000” card). Billboard’s feed items carry no image of their own, so the newsroom reads the page at card time, only for a story that becomes a card. It rides the source’s own breaker + limiter. Fail-open: a miss falls to the catalog/entity resolver, and desk_art.source still says which rung shipped. Health: the per-source ok rate on the Content data sources panel. No monitor: a miss costs one card its article photo, and the resolver still gives it art, so desk_art reason=no_art is the signal that matters and it is already watched. |
| music_ranking (RIAA boards) | cogs/music_desk.py, utils/credit_act.py | reason=credit_act, ok=True (CB-23, #4557): the Haiku act check (claude_api purpose=credit_act) overrode the old placeholder list on count credits a board can rank, RIAA cells and Billboard census parties alike; detail.not_act / detail.act name them (up to 10 each), detail.asked is the pass size. Expected near zero: on 30 days of real credits the model changed 1 of 1,138. A rise means a prompt or model change is dropping real acts; read the names. The existing music_ranking panel groups by reason, so it shows there. No monitor: a rare correction with no user-facing silence. reason=riaa_roster_partial (the Diamond boards wait: count acts of the roster read so far) | reason=riaa_timelines_partial (the fastest-to-Diamond board waits: count roads timed so far of expected roster Diamond singles, parse_json(detail)['expected'], #4794) | reason=riaa_diamond_list_partial (the Diamond singles and Diamond albums boards wait: the read of RIAA’s own Diamond list stopped short), ok=False — the RIAA record boards’ “absent over invented” rows; both trend to zero as the seven-day caches fill, and a count that stops rising is a finding. Measured 2026-09-03: #2862 put the genre loop between if complete: and the roster-partial else, so the row fired every slot (count=40 of 40) as a for/else; it is an if not complete now, with a test that records the emits. |
| riaa_fetch | utils/riaa.py | source (riaa), query (recent | timeline | tallies | artist | awards | genre | diamond), ok, duration_ms, count (rows parsed | history points | tally rows | act rows | awards naming the act), cache_hit (recent ~2h, timeline ~7d, tallies + artist + awards ~7d, durable); on a miss: reason (unreachable = no 200 | unparseable = a 200 with zero rows, i.e. the table changed; the site’s empty end-of-list payload on the page after the last one is the list’s END, not a miss (list_ended, a one-page act read as unparseable before 2026-09-02) | capped = an awards walk hit AWARDS_MAX_PAGES with pages left, so the list is not the whole catalog and is not served) + error. A timeline read of a BLANK record is ok=true with reason=no_history and count=0 (#4788): the site’s normal layout with an empty timeline, every date January 1, 1970, title/label/format empty (award 441124, a Diamond single on Zach Bryan’s list, measured 2026-10-25). It is cached for the timeline window and raises no error; before #4788 it was re-read every music-desk slot and raised 85 riaa unparseable (timeline 441124) errors over 2026-10-24..25. A 200 with no timeline container is still unparseable. query=awards reason=superseded (ok=true, #4794) = the music desk dropped one act’s cached award list because a roster Diamond single’s timeline was blank: RIAA replaced that award with a new record (re-certification), and the 7-day list still held the old id. Once per (award id, act) per process; the next slot re-reads the list. A steady stream of it for the SAME act means the replacement record is not in the act search either. A timeline read of a LATIN-program award is ok=true with reason=latin_program and count=0 (#2902), not a miss: RIAA’s Latin program has its own ladder (Oro / Platino / Diamante, 600,000 units a level against the main program’s Gold 500,000 / Platinum 1 million / Diamond 10 million), so parse_timeline deliberately maps none of it and returns an empty history – but the page itself is perfectly readable. That empty result used to fall through to unparseable, which meant a Latin award was never cached, cost a cold read out of the slot’s three-read budget on every attempt, and filed a recoverable riaa error each time, for a page that would never parse (measured 2026-09-03: award 453588, “DE LA NADA - REMIX (FT. DE LA ROSE)”, a Latin single certified that day). timeline_ladder tells the two apart, so unparseable goes back to meaning the one thing worth paging about: the site changed shape. The newsroom’s cert lane now also skips non-main-program awards outright, so the read is not requested at all — one read of RIAA’s own Gold & Platinum database (#2839, line B of #2800). The certification AUTHORITY read directly: measured 2026-08-20 to 09-01 the wire cert lane shipped 22 cards against ~180 awards RIAA issued (one in eight), and a Julia Michaels 2x Platinum reached no wire at all. recent walks the newest-first feed until a page’s oldest award is older than the caller’s window (the newsroom’s 3 days), capped at 8 pages of 30 (~15 awards land a day, 58 in one burst); timeline reads one award’s history so the take can state the previous level (count = history points, on a cache hit too). Rows for WATCHED/KNOWN acts become first-party SourcePosts (platform=riaa, handle riaa, relay_number=True) into the newsroom’s existing cert lane, so they classify, key (cert_story_key, <n>xplatinum|riaa) and dedup against the same certification from any wire. Gated on music_riaa_certs – graduated 2026-09-07 (master-only), so the gate is graduated_stage and resolves PRODUCTION in the master guild, OFF elsewhere (while it was a trial, anything but PRODUCTION routed its cards to #bot-logs whatever the parent newsroom’s stage, and such an audition never booked the settled cert key; audition=True can no longer occur). A recent read that stops short (a later page unreachable or unparseable) returns the rows it has, emits ok=false with that reason plus an error row naming the page, and is never cached; a timeline miss emits an error row too. One breaker (integration=riaa) + one limiter for the host. Fail-open (a miss → the cert lane keeps its wires) with emit_error(recoverable=True). Cloudflare refuses a bare Mozilla/5.0 (403) and accepts a browser-shaped user-agent; the client and the riaa probe send the same string, and the probe requires award rows in the body (a 200 challenge page without a table is a 502 on the sweep), so a future challenge page reads as a failure here and on the /debug/integrations sweep, never as a quiet cert lane. An RIAA audition’s #bot-logs header named music_riaa_certs as the flip that ships it (no longer reachable since the graduation). On the ops monitor as riaa in the misc integration health (_MISC_INTEGRATIONS, cache hits excluded) and on the data_sources dashboard panel. Expect ~12 recent reads a day per process (a 2h cache under the ~15-min desk tick) plus a timeline read per new watched-act award. diamond (#2915) is the Diamond boards’ own read and is NOT a roster read: it searches RIAA’s award database for every Diamond award of one format (awards[]=D), up to DIAMOND_MAX_PAGES (26) pages, held ~7d. Measured 2026-09-03 the singles list is 516 awards over 55 acts. Before this the two Diamond boards ranked only the 40 names of the streaming watch list, so they MISSED any Diamond holder outside it – the posted card omitted Post Malone (9, which is #2), Lil Wayne (6), Morgan Wallen (6) and Cardi B (4), and two X replies called the board wrong. A short read emits ok=false with capped, unreachable or unparseable and draws NO board (music_ranking ok=false reason=riaa_diamond_list_partial). The RIAA record boards (music_riaa_boards, graduated 2026-09-07 master-only alongside music_riaa_certs; utils/riaa_boards.py) add tallies (the top-albums list, 4 pages, ~7d cache) and awards (one act’s award SEARCH, every award naming the act, up to 8 pages, ~7d cache, at most _RIAA_TALLY_FETCH_PER_SLOT cold walks a desk slot – the diamond boards, the Platinum-singles board and the certified-units board all count features and collaborations off it, one read per act for every board); artist (one act’s awards-by-artist page) is parsed and cached but no board reads it today; a music_ranking ok=false reason=riaa_roster_partial row marks a slot where the diamond board waited for the roster (count = acts read so far), which is normal while a cold cache fills and a finding if it never clears. year_feed (#2871) is the YEAR board’s own read and is NOT a roster read: RIAA’s dated feed walked back to Jan 1, so the board ranks every act RIAA certified rather than the first 40 names of a streaming watch list. It is budgeted – one call takes at most YEAR_FEED_PAGES_PER_CALL (25) pages, resumes on the page it stopped on, and emits ok=false reason=filling until the walk reaches Jan 1, with music_ranking ok=false reason=riaa_year_feed_partial on the lane side and NO board drawn meanwhile (prefer absent: a partial walk under-counts every act). Measured end to end 2026-09-03 the cold walk is 61 pages / 1830 rows / ~9 minutes; a covered walk then tops up from page 1 every ~6h and costs one or two pages a day. It REPLACES nothing – the roster’s awards searches still run for the Diamond, Platinum-singles and certified-units boards, so the year board used to ride them for free and now pays its own way. Held a week (_YEAR_FEED_TTL), that is ~13 reads a day against the roster’s ~16, i.e. about a 45% ADDITION to the RIAA read budget. A 1-day TTL was the first choice and cost ~64 a day, because the row expiring forces the whole cold walk again. Rows merge by award id with the later certification date winning, so a re-certification upgrades its row instead of counting the title twice. Caller attribution (#2937): every riaa_fetch row a scoped caller causes carries surface + path (<surface>.riaa_fetch.riaa). The timed reads stamp through timed_event; the cache-hit and per-alias rows are plain emits and stamp through instrument.surface_fields (riaa._emit_fetch). Before this, 371,701 rows in 7 days had surface=None, so a failed read could not be traced to its lane (#2902 went to the wrong lane). A row with no surface now means the CALLER has no @scoped_surface, not the client. Both callers are scoped on main (#4098): 0 of about 432,000 rows carried surface=None from 2026-10-16 to 2026-10-25. Panel riaa_by_surface on the media board. reason=omitted_credit_lead_short (ok=false, no duration) is a FETCHING awards_for_act read whose search for an omitted-credit LEAD came back short; the act’s catalog still serves without that lead’s rows. A cold lead under a cache-only (fetch=False) read emits nothing and returns None, so the caller makes the fetching read (#4796). Before that fix, a cold “young money” lead emitted this reason 742 times in 36 hours with no request made, and it was the whole of the ops monitor’s riaa fail rate. |
| music_desk_career | cogs/music_desk.py (#3363) | chart (hot100|billboard200), kind (entry — a settled debut’s ordinal | projected_entry — the standing a HITS-projected debut WOULD set, #3363), matchup (title - artist), count (entries on the act’s Billboard chart-history page) — the chart lane attached a grounded CAREER STANDING (“her seventh entry on the Billboard Hot 100”) to a row whose own story is thin. Why it exists: measured over 1,034 of our posts (2026-09-08..15) a bare chart position is the worst-performing type on all 45 wire accounts we read (0.69x their own median) and 37% of our output, while a career standing is the best (1.30x) and 1% of ours. Why it is narrow: the A/B improved 4 of 4 debut/re-entry rows but REGRESSED a row that already had a hook (Luke Combs’ “a new peak 24 weeks into its run” 0.92 → “his 43rd career entry” 0.88), so it fires only on debut/reentry, and only when music_news.is_latest_entry confirms the title is the act’s newest (an ordinal on an older title implies a recency it lacks — Ariana Grande’s 99th of 110). Emitted only when a fact was produced, so the rate is how often the clause is available; fail-open, every miss ships the card unchanged. projected_entry is the forward twin on the projection lane: it swaps that framing’s flat career ban for the one grounded fact. The ban was not producing silence about a career — measured 2026-09-19 it produced a WRONG one, “a solid new entry with no chart history to lean on yet” about Dolly Parton, who has 51 Billboard 200 entries. Its guards refuse a title already on the history page (a catalog re-entry, not a debut) and an empty page (a guessed slug that 404s reads the same as a genuine first-timer) |
| pollstar_fetch | utils/pollstar_charts.py | source (pollstar_charts), query (the chart key: global_concert_pulse|live75|new_tours|artist_power_index|mediabase_<format>, plus a <key>:issue row per ISSUE RESOLUTION — #3357, the config read that finds the current week’s chart id, kept separable from the body read so one failing does not hide the other; it carries folded detail.issue_id), ok, duration_ms, count (rows parsed), cache_hit (~6h durable, keyed by ISSUE so a new week busts the body cache at once); on a miss: reason (unreachable|unparseable|no_issue — the config did not parse, so the read falls back to the seed issue and the freshness fence takes it from there) + error — one Pollstar free-tier chart read (#reporting-oversight). The live-music numbers Billboard does NOT publish (worldwide active-tour ticket averages, avg boxoffice gross/city, new-tour on-sales, 24 Mediabase radio formats), off the public chart JSON at data.pollstar.com, anon-only — no auth is ever sent, and only the free-preview rows the endpoint returns are taken. ToS one-way-door (owner-accepted, epic issue): Pollstar’s terms forbid redistributing chart content and forbid stripping attribution, so every posted number MUST be attributed to Pollstar (PollstarChart.attribution + the chart link). Breaker integration=pollstar_charts. Fail-open with emit_error(recoverable=True); DARK until a desk consumes it |
| artist_news | cogs/music_news.py | guild_id, count (synthetic candidates produced this slot), + folded detail{checked, hits, stale, top, tail} — one PROACTIVE watched-artist news pull ran (owner steer 2026-08-20). The newsroom’s reactive feed only reports what a followed account posts; this lane inverts it: each slot it rotates a bounded set of the top watched artists (top ~25 fast, the ~75 tail slow — the split is detail.top/detail.tail), asks Perplexity for each one’s biggest recent RELEASE / CERTIFICATION / TOUR update, and feeds any answer through the SAME classify → tier → verify → compose → gate pipeline as a synthetic SourcePost (platform=artistpull, source tag artist_watch, which may_relay_number reads as untrusted so the pull story ships ONLY when independently verified). checked is the artists swept, hits the ones whose pull returned FRESH real news, stale the ones dropped because the model’s stated event date was old or missing (#2488 — the pull’s recency gate; the pull has no source-tweet time, so a now-stamped synthetic post would otherwise defeat the _RECENCY_HOURS window, which is how a nine-month-old $1B tour record shipped as “breaking”). (count then trails hits after the seen-set dedup and the _ARTIST_PULL_MAX_STORIES slate cap.) Gated on the music_news_artistpull experiment (STAGING default; rides the parent newsroom’s staging-vs-production routing like the other overlays). count sitting near 0 while pplx_music_news stays green is the COMMON case — most swept artists have no news that sweep, which is correct; checked collapsing toward 0 while kworb’s chart_fetch stays green = the watchlist build broke. STREAMING milestones AND chart positions are dropped here (the desk owns both) — see music_news_filtered reason=artistpull_desk_owns. A SALES-unit milestone is a known gap (no ladder, no discovery, no dedup key), tracked as #2793. The pull’s shipped stories are normal music_news_scored/music_news_posted rows with the artist_pull lane tag. detail.mode distinguishes the two proactive lanes that emit this event: pull (Perplexity, certifications/tours) and newrelease (Apple catalog, album drops, owner steer 2026-08-21). The deterministic new-release lane (_gather_new_releases) reads each rotated watched artist’s Apple Music discography (keyless, free — a deterministic read the metered pull can miss; the top ~25 swept every slot are caught for sure, the tail best-effort), turns an ALBUM (singles/remixes filtered) dated within _NEW_RELEASE_WINDOW_DAYS into a release candidate with a STABLE newrelease:<collection_id> id (a drop posts once), and rides the SAME pipeline; gated on the music_news_newrelease overlay (OFF default — dark until a mod opts in, since it rides the parent stage for routing). Apple OWNS releases (the pull no longer asks for them). checked/hits read the same for both modes; count near 0 is the common case (a watched artist rarely drops an album in a given week) |
| artist_watch | cogs/music_desk.py | guild_id, count, detail{checked, tracks_windowed} — one TOP-ARTIST WATCH milestone sweep ran (owner steer 2026-08-04): the watched artists’ free kworb Spotify metrics (career streams, monthly listeners, cumulative totals for their tracks on the Global Daily chart) through the SAME songstats_milestone_state rungs as the opportunistic Songstats checks, so the two detectors share one high-water baseline and can never double-fire. count is crossings composed this sweep (usually 0 — rungs are sparse by design), checked the rung checks run, tracks_windowed the watched charting tracks eligible for the bounded, hour-rotated track-page window. The watch’s other halves (lower chart/radio floors, the wider callable band for watched artists) ride the existing lane events — a watched chart move is a normal music_desk_scored phase=chart row. The sweep going silent = the watch is dark (kworb miss / empty watchlist); checked collapsing toward 0 while kworb’s chart_fetch stays green = the watchlist build broke detail.mode=tenure (#2800 line C) marks the CHART-TENURE pass on the same event: checked = watched rows on the Spotify daily charts that carry days, hits = rows at or past the first rung (100 days), count = tenure cards composed. hits near 0 while the chart sweep is healthy = the watchlist has no long-running songs, which is normal for a young roster; checked at 0 while watch_chart cards flow = the days column stopped parsing. |
| milestone_source | cogs/music_desk.py | surface=”music_desk”, matchup (the artist), source=”kworb”, ok (do kworb and Songstats land on the SAME career rung? null when Songstats returned nothing), reason (resolved | no_row | no_total | feed_out), detail{metric, kworb, songstats, gap_pct, matched} — one CAREER-STREAMS rung check resolved its source (_kworb_career_total, 2026-09-14). Why it exists: the career rung is the ONE milestone metric on which this repo’s two Spotify sources disagree. Measured live over 14 top acts: monthly listeners agreed to 0.00% on 8/8 and track totals to ~0.1%, while career totals ran 0.0%–4.3% apart with Songstats always higher, and 2 of the 14 sat in the gap band where Songstats clears a rung the kworb board does not. That gap shipped the Weeknd “crosses 100 billion career Spotify streams” card on 2026-09-14 — Songstats read 102.1B, kworb 97.9B, and the card’s own standing strip (kworb) printed #4 with 97.9B beside the 100B hero. The Songstats artist total also RESTATES DOWNWARD, which a high-water rung cannot represent (Bad Bunny posted “at 133.3B” on 09-13, the same field read 130.7B a day later). kworb now drives the rung alone and this event keeps the divergence visible after it stops driving the break. ok=false is the useful series — it counts the checks where the old path would have fired early, so a rate near zero means the sources have converged and a rate climbing means the gap is widening. A gap_pct turning NEGATIVE (Songstats BELOW kworb) or widening well past ~5% means kworb has started lagging and the source choice needs a re-look. reason is the funnel. no_row means no EXACT name match on the board. That covers an act the board does not carry AND a COLLAB credit: kworb’s strict match is a CREDIT test — right for a rank, wrong for a career total — and live it hands back Drake’s row for “Drake & 21 Savage” (his solo 141.0B) and Future’s for “Future & Metro Boomin”, so this path does not use it. detail.matched carries the board’s CANONICAL name, which is also the rung’s entity key: it is shared with _watch_milestone_units, and folded matching makes the spellings differ often (“Beyonce” → “Beyoncé”, “JAY Z” → “JAŸ-Z”, already in the live table as artist:jaÿ-z), so keying by the news subject would be a second state row over one figure. feed_out is the one to alarm on: the all-artists board came back EMPTY, which stops EVERY career milestone, so a run of it is an outage rather than a quiet day. no_row is an act off the board, normal below the ladder’s 50B floor. Panels: “Career-streams rung: how the kworb figure resolved” and “Career-streams source gap” on tootsies-desks. The event absent entirely = kworb is unwired, and NO career milestone can fire |
| album_milestone_sweep | cogs/music_desk.py | guild_id, count, detail{pages_read, ytd_rows, crossings, stale, undated, joint} — one ALBUM milestone sweep ran (_album_milestone_units, owner ask 2026-09-30 “both sales and streams”). Two halves: a rotated window of watched acts’ kworb ALBUMS pages feeds the album_streams ladder (lifetime Spotify, global; aged at the album’s daily by the same freshness gate as the per-track sweep), and the HITS year-to-date albums chart feeds album_units + album_sales (US) for albums released THIS year only. count is cards composed and is usually 0. Read pages_read and ytd_rows for health: pages_read 0 for a day means the kworb half is dark; ytd_rows 0 for a week means the HITS read failed (see hits_fetch), or it is early January. Graduated at ship (owner steer 2026-09-30, no trial): graduated_stage, live in the master guild; any other guild emits nothing. Since 2026-09-30 detail.joint counts crossings credited to more than one act (the iTunes credit, each act confirmed on its own kworb page). Panel: album_milestones. |
| catalog_sweep | cogs/music_desk.py | guild_id, count, detail{read, readings, roster, mode} — one CATALOG-COUNT sweep ran (detail.mode = total or daily, #2800: the same page counted twice, once per column; one event each) (#2772, owner question 2026-08-31 “why didn’t we proactively find different stats about artists and genres?”). Every other music lane fires on a printed chart week or on one scalar crossing a round number; this one asks the CORPUS a question. Each slot it rotates a bounded window of watched artists (MUSIC_DESK_CATALOG_FETCHES, default 6), reads each act’s kworb SONGS page, counts the rows at or above each rung in catalog_stats.CATALOG_RUNGS (1B and 2B), and upserts the counts into artist_catalog_counts — the GLOBAL, cross-guild field a rank is taken over. A count that TICKS UP breaks through the shared songstats_milestone_state rungs as metric catalog_count, so seed-on-first-sight holds and a rung breaks once. read is pages fetched this sweep, readings the BOUNDED counts stored, roster the watched acts carrying a kworb artist id. count near 0 is the NORMAL case — a song crossing 1B or 2B is rare, so the sweep’s job most slots is to fill the corpus, not to post. That makes count useless as a health signal: read read (0 = the sweep is dark, a kworb miss or an empty watchlist) and readings (collapsing while read holds = the songs pages truncated above the rung, so count_catalog marked the counts unbounded and none may be stated). The corpus ageing out is the quiet failure: catalog_counts_at_rung drops rows older than 30 days, so a sweep that stalls silently shrinks the ranking field until coverage_ok (90%) gates every rank — and because the lane refuses to compose without a ranking clause, the lane then goes SILENT rather than shipping bare counts. That is deliberate (a bare count lets the compose invent a record) but it means a stalled sweep looks exactly like a quiet one on count alone; read and readings are what separate them. The ROUND-COUNT path (owner ask 2026-09-10): a TOTAL count landing on a multiple of five with no rank clause goes to the candidate with the web qualifier REQUIRED, so its outcome is NOT in this event – read qualifier with phase == 'catalog_stat': verified is a card in flight, required_missing is the lane finding no reported ranking (detail.settled true = the rung advanced, no retry; false = a transient miss, retries next slot). Measured before the path shipped (2026-09-10): 224 total-mode sweeps in 14 days, 0 cards. Gated on the music_catalog_stats experiment (STAGING default) Since 2026-09-18 the TOTAL-mode event also reports the PER-TRACK half (owner steer, off the Hip Hop All Day “Plot Twist” post): detail.crossings is the number of lead rows on the pages read whose lifetime total newly crossed a per-track streams rung (100M, 250M, 500M, 1B…) since the last read, and detail.track_cards the plain milestone cards composed off them (capped by MUSIC_DESK_MAX_CATALOG_TRACK_MILESTONES, default 2). The state is the SAME songstats_milestone_state track: row per song the Global Daily sweep uses, read and seeded in bulk (db.milestone_rungs_bulk / seed_milestone_rungs_bulk), so the FIRST read of every page seeds ~500 rows and shows crossings 0 by design; after that, measured on the top 30 acts, about 30 crossings a week are due (18 at 100M) against the 2 the Global Daily chart could see. A delivered per-track card also writes the global stream_crossings ledger (_record_ship), the stream REPORT lane’s input (row below). Since #3467 the TOP TIER is read every slot: the first MUSIC_DESK_CATALOG_TOP_TIER (10) acts of the kworb ranking are read on every sweep, outside the rotation, and the rotation covers the rest; detail.read is every page read this sweep (top tier plus the rotated window) and detail.top_tier the top-tier count, so a crossing on a top act’s page lands within the slot kworb shows it rather than within the day. Since 2026-09-19 a crossing must be FRESH to be a card: milestones.crossing_age_days measures how long ago the total crossed the rung at the row’s own daily pace ((total - rung) / daily); older than MUSIC_DESK_FRESH_CROSSING_DAYS (7) advances the rung with no card (db.advance_milestone_rung), emits music_desk_scored with reason=stale_crossing (phase milestone, detail.age_days / detail.rung), and counts in detail.stale. Why: the rung is measured between READS, and a track: row the chart sweep seeded can go unread for months, so the songs-page sweep’s first read shows a REAL crossing that is old news – live 2026-09-19, “Shape of You” 5B (66 days old) and “Wonderwall” 3B (29 days) both posted as “just passed”. The same gate runs in the Global Daily sweep (_watch_milestone_units), on an actual crossing only. Since the second fix (2026-09-19, the “That happened 40 days ago” reply) a row with NO daily figure is not taken as fresh: it is DEFERRED – no card, no rung advance, a music_desk_scored drop with reason=undated_crossing (detail.rung / detail.entity_key), counted in detail.undated; the next read that shows a daily decides. kworb’s daily column is blank for whole pages at a time (Arctic Monkeys 0 of 143 rows that day), and “Circles” at 3.28B shipped as “reaches 2 billion” through that hole. The backlog itself came from the chart sweep seeding track: rungs off the track page’s chart-history Total (500M for a 5.1B song); it reads the songs page now. stale high on a page’s first re-read is the expected drain of that backlog; stale steady at zero afterwards is normal. undated steady above zero for an act for days = kworb is not publishing dailies for that page, and its crossings wait; undated_crossing drops name the songs. Both markers (stale_crossing, undated_crossing) are in the ops monitor’s NON_SCORE_REASONS: score=0.0 stamps from a lane that composed nothing, never a judge verdict (unregistered, the 29 settles of one 2026-09-19 sweep read as 29 milestone posts scored 0.0). |
| desk_art | cogs/music_desk.py + cogs/music_news.py + cogs/cinema_desk.py + the three WIRE desks (cogs/pop_desk.py, cogs/sports_desk.py, cogs/cinema_news.py) | surface, ok (art bytes returned), source (the rung that answered: artist_photo | discography | album_search | song_search | release | release_discography | catalog_cover | artist_portrait | tmdb_poster | deezer_artist | deezer_artist_exact | wiki_logo | wiki_portrait | steam_art | …). wiki_portrait is new (#deezer-403) — the Wikidata/Commons artist photo, the SECOND provider under Deezer’s artist picture. Until it existed every artist photo in the repo came from Deezer alone, and on 2026-09-14 Deezer began answering 403 to Railway’s IP: a music card whose title the Apple catalog does not carry (a new radio single is the normal case) then had no rung left at all and shipped bare, which is the David Banner “+8” most-added card. Read its SHARE the way artist_portrait’s is read, and with the opposite sign: wiki_portrait is a FLOOR, so a low steady share is health and a share that climbs says Deezer has gone quiet again. steam_art is new (#2741) — the Steam catalog rung for a VIDEO GAME subject, which the wiki_image rung structurally could not serve (pageimages excludes the non-free box art every game article leads with, so a game resolved NOTHING and the card fell to the branded floor). Watch it the way artist_portrait is watched: steam_art appearing where a game subject used to produce none is the fix working, and a game subject falling back to none again is the signal that Steam’s storesearch leg has gone quiet — a silent, fail-open degradation that raises no exception (the steam integration probe is the other half of that watch). music_desk’s single catalog label split into catalog_cover vs artist_portrait (#2274) when resolve_art_url gained a portrait floor: a cover is the record, a portrait is a stand-in for a record the catalog could not supply, and one label for both would let a rise in portraits read as healthy art coverage. Watch the portrait SHARE — it is the signal for a cover-lookup regression, which is otherwise silent because the card still ships a picture. It was HIGH on the song lanes until #2278 fixed the subject-kind title guard; live A/B moved 24 of 45 song rows from artist_portrait to catalog_cover, so the expected steady state is now mostly covers, with portraits only for a release the catalog genuinely lacks. A fourth cover rung reads the album’s TRACKLIST, and a split-credit hit reports catalog_cover_split (#3508 / #3511): resolve_art_url reads the lead act’s newest discography tracklists (apple_music.artist_tracks) for a SONG subject after the keyword and album-discography rungs miss, so an album CUT on a record the search does not index yet resolves its album’s cover instead of a portrait. A hit on the WHOLE credit (an exact iTunes artist id for the full credit string) reports catalog_cover as before. A hit reached by SPLITTING the credit (“Yeat” for “Yeat & Drake”, the lead of “Lead feat. Guest”) reports catalog_cover_split, which the delivery media gate judges like any unverified image (it is absent from _IDENTITY_VERIFIED_ART). Read the two shares together: catalog_cover_split rising while media_coherence mismatches on music_desk stay flat is the rung finding real cuts on joint credits; catalog_cover_split rows that the gate DROPS (media_coherence reason=mismatch with source=catalog_cover_split) are the wrong-cover cases the label exists to catch, and a steady stream of them says the split is finding the wrong act. A music_desk or x_mentions media_coherence row can now come from the ROOM path as well as the crosspost (gate_split_art, #3515): a split cover is judged once before the room copy ships and, if it passes, again by the crosspost, so one split card can leave two rows. A portrait share climbing back toward the covers is the regression signal. A FIFTH cover value, catalog_cover_fuzzy (2026-09-20, the Miguel Newberry cover): the rung is now EARNED per row off its credit (catalog_art.cover_rung). A cover whose row matches the request EXACTLY – the credit deterministically (credit_verified: whole billed acts, no act the request never named) and, when a title was requested, the core title equal (title_verified: no fuzzy containment, no ratio, no remix or live tag the request lacks) – reports catalog_cover as before; one that matched only by the spelling ratio or the fuzzy title match, a billing with an extra act (a remix or joint single), a row with no credit, or the artist-level no-credit last resort (pick_any_art_url), reports catalog_cover_fuzzy, which the delivery media gate judges by face like any unverified image (absent from _IDENTITY_VERIFIED_ART). The card that forced it shipped Miguel Newberry’s same-title single cover on a Miguel Billboard 200 card: the matcher’s substring hole let the credit pass, the rung said catalog_cover, and the exemption overruled the judge’s correct flag (media_coherence reason=verified_source verdict=subject). Read the two together: catalog_cover_fuzzy should be a SMALL share (a stylized spelling on one side, “Rick Ro$$” for “Rick Ross”), and a media_coherence DROP on it (reason=mismatch, source=catalog_cover_fuzzy) is the gate catching a wrong cover the credit rule let through — the exact event this rung exists to make visible. A fuzzy share that climbs says a source has started spelling acts differently from the catalog, which was silent before because every cover wore the exempt label. entity_image’s music cover now also reports deezer_cover when Deezer answered (it said catalog_cover for both providers before). The desk_art panel groups by source dynamically, so it picked the new values up with no axiom_setup.py change, reason (on a miss: no_url = the resolver found no URL | download_miss = a URL resolved but the image GET failed | no_art | no_subject | error). music_desk’s _art_bytes now names which of the two it was (#deezer-403); it collapsed every miss to no_art, so the David Banner card could not be told apart from a CDN blip without replaying the whole ladder by hand. Its sibling _fetch_art always split them, and the two now match, phase (take_retry on the wire desks only) — a desk’s ART resolution resolved or missed. The art path used to be silent, so an art-blank card (“why didn’t it have art”, owner) could only be spotted by eye: a *_posted carded flag says a card was BUILT, never that it had art. The music newsroom now emits it for EVERY story, including the ones with no number (#wrong-missing-images): it used to return before resolving art whenever the story had no figure, so the whole RELEASE lane resolved no art and emitted no event — the lane’s art misses were invisible, and its posts shipped imageless in the room and on the timeline. Expect a small rise in surface == 'music_news' volume from that lane alone; a release story whose art misses is now a countable ok=false instead of silence. The three WIRE desks emit it only for their LAST rung (phase=take_retry): they resolve art BEFORE compose on the wire HEADLINE, and a headline that names no entity (a chart/roundup title) classifies to an empty subject list, so that pass has nothing to resolve — the composed TAKE is then the first text that names a subject with art, and the retry runs on it. Query phase == 'take_retry': ok=true counts posts that would have shipped bare, ok=false counts the ones that still do (the residual, also visible as the *_posted source=none rate). cinema_desk’s bare-text lanes now carry count (#individual-art) — the number of posters resolved for the take, one per show it names (the release “what’s out” roundup resolves several; a single-title news/awards take resolves one). count > 1 is the multi-poster path; count == 0 with ok=false is a lane that resolved no art at all. music_news also carries tour (#tour-announcement, folded into detail) — whether the card took the TOUR path (_claim_is_tour or _is_tour_subject). It is the queryable half of that fix: tour == true with source == 'deezer_artist' is the intended photo-first result, while tour == true with source == 'catalog_cover' is the BACKDROP net (no artist photo resolved, so the act’s own cover keeps the card). A backdrop share that climbs says the artist-photo rungs have gone quiet, which is silent otherwise because the card still ships a picture. music_news also carries bare (folded into detail, 2026-09-14) — a TRI-STATE: true shipped the picture with NO card drawn over it, false was a candidate for that rung and kept its card, and ABSENT was never a candidate. One rung takes it: a figureless TOUR announcement whose art is feed_photo, the image the publisher filed WITH the story, AND whose image the image_is_tour_admat judge says lists shows. It records what SHIPPED, never what was attempted (Codex review on #3291): the judge can say no, and download_image_bytes validates only the HTTP status and the body length, so a non-image body reaches build_bare_art_card and comes back None — in both cases the branded card ships, and a bare=true row there would be a lie told by the one signal this rung has. So read false as the judge’s refusals: a run of them is how a judge that has started refusing everything becomes visible, which is otherwise silent because the card still ships. Read it against delivered == 'room' (Codex review on #3291): the card is built before _sched_deliver branches on the stage, so a STAGING audition runs the same route and would otherwise inflate the shipped-bare share with posts that only reached #bot-logs — the same dimension desk_link carries, for the same reason. That image is the tour admat, so it already prints the tour name, the act and every date; the card repeated all three in its headline and its bottom scrim covered the date list (owner steer, the Malcolm Todd “Do That Again Tour” post — “this probably looks better without our markup on it”). Read bare == true as a SHARE of tour == true, never as a count: the rung is rare because feed_photo itself is rare. A bare share that climbs past the tour lane says the feed rung started answering stories it did not answer before, and those posts now ship unbranded — no wordmark, no tag, no source stamp — which nothing else in the ledger reports. music_news also carries dropped + reject_detail (both folded into detail, 2026-09-16) — a feed image the GEOMETRY guard refused before it could become the card. dropped names the rung (feed_photo) and reject_detail the band and the size (undersized 728x90, banner 970x250). The case they come from: Billboard’s College Campus Tabby Tour story opened its body HTML with a 728x90 house ad, industry_feeds._first_image took it as the article’s photo, and it shipped to @tootsiesbar as the art. Nothing downstream could catch it — download_image_bytes checks only the HTTP status and the body length, and the delivery coherence gate (x_crosspost._media_gate) passes a branded graphic BY DESIGN so our own chart and number cards clear it, so an ad, which names no person and no work, contradicts no take and its wrong-SUBJECT question cannot see it. catalog_art.ad_banner_reject answers it on GEOMETRY with no vision call: display advertising is sold in fixed IAB units whose shapes no editorial photo has (the leaderboard past 8:1, the skyscraper 1:3.75), and the same 200px floor catches a feed THUMBNAIL (Billboard’s own lead images are 237x158 crops). These two fields exist because the guard degrades SILENTLY otherwise: a refused image falls through to the catalog/entity resolver, so the story still carries art and ok stays true — a guard that had started refusing every real photo would look exactly like a healthy lane. Read the share against a flat feed_photo count, and read reject_detail’s band: a run of banner is the ad class it was built for, while a run of undersized on a feed that used to answer says the publisher moved to thumbnails. What geometry CANNOT catch is the 300x250 medium rectangle, which sits inside both bands; that residual is deliberately left open rather than answered with a vision judge, since the rung builds about 4 cards a week and the moment-framed photo gate rejects a REAL Billboard lead photo as readily as the ad (5/5 each, measured on the Tinashe story) tour + event_shape (#3413): tour is the bare tour flag the card build used to derive from phrase lists; event_shape is the classifier’s announcement lane on a release (record / tour / tour_update / live / nonaudio), a fold field (parse_json(detail)['event_shape']). Before it a LIVE moment was stamped tour=false while taking the tour art path, so the dashboard could not tell live from release; group music_news art by event_shape to read the four lanes apart. |
| release_art_stale | cogs/music_news.py | surface (always music_news), source (the rung that answered INSTEAD — a portrait rung, or none when no art resolved at all), reason (the REFUSED row’s release date, or undated), detail.topic (the story subject) — a RELEASE card refused a matched catalog row because the row is from a different era than the story, so its cover would have claimed the wrong record (_row_is_a_different_record). The case it comes from (#3171): Linkin Park’s “Faint [Live in São Paulo]” dropped 2026-09-11 off the UNSHATTER film soundtrack, the wire named only “Faint”, and resolve_music_row answered with the 2003 Meteora recording — catalog_art._rank_rows prefers the clean original over a live edition when the requested title does not name one, which is right for a cert or chart story and wrong for a drop. The card then stamped NEW RELEASE over a 23-year-old cover. The same shape shipped the 2006 “IRREPLACEABLE” cover on five B’Day 20th-anniversary cards and the 2009 “Fear” cover on “Fear of Missing Out”. Read source, not just the rate. The guard trades a wrong cover for the act’s PHOTO, so a healthy steady state is mostly portrait rungs; a rising source == 'none' share means it is trading covers for BARE cards instead, which is a worse post than the one it replaced and the signal to revisit the window (_RELEASE_ROW_MAX_AGE_DAYS, 365 days). Measured before shipping: over 30 days, 19 of 71 catalog-matched release cards would fire this, about 13 of them plainly wrong art. A rate far above that says the window or the throwback carve-out (retrospective.states_past_interval) is mis-calibrated, not that the bug got worse A desk-flagged THROWBACK never emits it (2026-09-17): the guard reads the CLAIM for a stated backward interval, and the claim is the classifier’s extraction, which strips the time frame – “15 years ago today, Adele’s ‘Someone Like You’ hit No. 1” classified as “No. 1 on the Hot 100, first #1 featuring only piano and vocals”, so the guard refused the correct 2011 cover, the card fell to a Deezer namesake’s portrait, and the media gate dropped that (a bare number card on X). _artwork_and_link now takes the desk’s own retro flag (composed.meta["retro"], the same flag the link route reads) and skips the guard for a throwback. A daily stream count never emits it either (#3296, 2026-10-15): a claim mn.is_daily_stream_count matches (“6,906,506 streams on Spotify yesterday”, popstats02’s album sweeps, kind chart) is about the existing record, and 34 of the 60 firings in the week of 10-08 were this shape, each trading the right album cover for the artist’s photo. A chart story about a title already on the chart never emits it either (#4131, 2026-10-16): on the chart kind, a classifier chart_event of hold / move / none / exit with no edition word keeps the row (mn.chart_story_is_same_record). Replayed on 30 days of firings, it removed 16 chart-position firings. A debut, a re-entry and an edition word still emit it. |
| tmdb_calendar_filter | utils/tmdb.py (releases_between) | source (tmdb), op (discover_upcoming), count (rows KEPT), query (the window, <start>..<end>), + folded detail{dropped, undated, stale} — the coming-soon calendar DROPPED rows TMDB dated outside the window it was asked for (#3285). dropped is the stale-dated rows; undated is rows TMDB gave no date at all, counted apart because the two faults are different and one number for both reads as noise. TMDB matches a film on any US theatrical date in the window but returns the film’s PRIMARY date, so a re-release comes back in-window carrying its original date: Avengers: Endgame dated 2019-04-26 led the live 2026-09-15..2026-09-28 query, reached the card under the hero “Opening Next”, and the take called a 2019 film the most anticipated of the fortnight. The correct date is not in the response, so the row cannot be dated and is dropped — absent beats invented. This is a SILENT, fail-open drop: it raises nothing and only shrinks the board, so this event is the one place it shows. Emitted on EVERY build, not only on a drop: count is the starve signal, and a starve does not need a drop — TMDB returning six in-window rows is the same blackout as TMDB returning twenty of which fifteen are stale, and a drop-only event is missing exactly then. Steady state measured over five consecutive 14-day windows is 0–3 dropped against 17–20 kept, for a card that draws 10 and needs 5. WATCH dropped climbing while count falls: that is the calendar starving, and below 5 kept the board goes absent with no other signal. READ IT PER BUILD, never summed across a bin: the 5-row floor is a per-build threshold, so two starved builds that kept 3 rows each sum to 6 and read as healthy. The panel graphs min(count) (the bin’s worst build) against max(dropped), with count() alongside as the cadence |
| tmdb_poster_verify | utils/catalog_art.py (_verified_poster_url, #2759) | source (tmdb), op (search_movie | search_tv), query (the requested title, truncated to 120), reason (verified | same_work | title_mismatch | year_mismatch | judge_failed | no_match), + folded detail{reject_detail (TMDB’s title on a refusal; <title> (<year>) on a year_mismatch), answer (TMDB’s title on a same_work accept), via_model (true when the Haiku same-work judge decided, #4174)} — ONE film/TV poster title check. TMDB’s search answers any query with its top hit, so a wrong name upstream got a real poster of a different film: the classifier’s “Looney Tunes” for “Coyote vs. Acme” fetched the 2003 “Looney Tunes: Back in Action” poster, under the catalog label the delivery media gate trusts. The rung now requires the titles to be EQUAL after case, accents, &/and, punctuation and a leading article are folded (a show request may end in “Season N”, which is dropped). Since #4174 a title mismatch is judged, not refused outright: claude.same_screen_work (Haiku, purpose=screen_same_work on claude_api, cached 30 days) answers whether TMDB’s hit is the same work under another name -> same_work (ships) or title_mismatch with via_model (refused). judge_failed is an error or an unreadable answer: refused, fail closed, not cached. title_mismatch WITHOUT via_model means no Claude client was attached. year_mismatch is a bracketed request year more than one year from TMDB’s (a same-title remake), refused with no model call. A refusal returns no poster and the card drops to its next rung, so it is otherwise silent. Panel tmdb_poster_verify (Media board). No monitor: the rung builds about 10 cards a month (desk_art source=tmdb_poster plus market_subject, 30 days to 2026-10-15), too few for a rate to mean anything, and a refusal degrades one card to its next rung rather than going dark. Read a rising title_mismatch share on the panel instead. |
| catalog_title_confirm | utils/apple_music.py (confirm_band_rows, #3440); call sites in resolve_apple_music_url, _resolve_via_artist, song_release_date and deezer album_release_date, track_release_date, album_genres, album_top_track. Also utils/catalog_art.py (_judge_release_names, #4115) | source (itunes | deezer | itunes_release_name: the #4115 bracket-name check in resolve_music_row, one event per UNSURE candidate name, Haiku via classify() purpose catalog_release_name; there different drops the name, same_work keeps it, and judge_failed KEEPS it (the old rule), so a judge outage shows as a rising judge_failed share and never as missing covers | itunes_credit | deezer_credit: the CB-07 credit check apple_music.judge_credits, one event per ratio-only iTunes credit or overlap-only Deezer credit, Haiku via classify() purpose catalog_credit_match SAME / DIFFERENT, cached 30 days per (request, credit); different and judge_failed drop the row), query (the request, truncated to 120), reason (sure_present | same_work | different | judge_failed), count (distinct band titles), + folded detail{via_model, cached, answer (the kept title)} — ONE ambiguous-band decision. The fuzzy title matchers (_title_matches, album_title_matches) accept a title by ratio 0.6 or by containment. A pass that is not equal (CB-07: at any ratio; #3440 cut it at 0.85) or, for an album, adds a new core word is the BAND. sure_present = an equal row in the same result set beat the band rows, no model call. Otherwise claude.match_catalog_title (Haiku, purpose=catalog_title_match on claude_api, cached 30 days per request+candidate pair) picks the same title from the real rows or none -> same_work (kept) or different (dropped). judge_failed = an error or an unreadable answer: band rows dropped, fail closed, not cached. No event when no row is in the band. Panel catalog_title_confirm (Media board). No monitor: the dry run on 90 real chart rows reached the band 25 times and the model 3 times, so a rate is too thin to gate; a refusal drops one link or date to absent rather than going dark. Read a rising judge_failed share on the panel. kind=scope (#2784): a fuzzy “alive” / “living” phrase the exact ranking regex does not read (output_checks.universal_scope_candidates, “the best drummer alive”) goes to Haiku claude.universal_scope_claim (purpose=universal_scope, CLAIM or MOOD). claim and judge_failed hand the phrase to the ranking arbiter above as an ungrounded claim (the drop direction); mood ships it. 0 of 10,725 real texts produced a candidate (2026-10-17), so expect about 0 rows; a scope row is a line worth reading. |
| wiki_namesake_check | utils/music_markets.py (_pick_namesake, #3319), called by _wiki_portrait, the Wikidata rung of resolve_art_url | source (wikidata), query (the lead act name, truncated to 120), reason (picked | unclear | judge_failed | no_judge), count (same-name music acts with a photo), + folded detail{answer (the picked QID, or “”)} — ONE namesake decision. wikidata.artist_portraits returns every music act with a P18 photo under the lead’s name; one act needs no call and emits nothing. Two or more is the Steve Lacy case (the jazz musician Q504641 ranks ahead of the R&B singer). Haiku reads the card’s credit, title and each act’s Wikidata description (classify, purpose=wiki_namesake on claude_api, cached 30 days per lead+credit+title+QIDs): a number picks that act (picked); UNCLEAR gives no photo (unclear, the card drops a rung). judge_failed (an error or an unreadable answer) and no_judge (scripts, before boot) keep the old first-match rule. Panel wiki_namesake_check (Media board: picked / unclear / failed, plus failed_pct). No monitor: the Wikidata rung answered 0 cards in the 14 days to 2026-10-17, because it runs only when Deezer misses. Read a rising failed_pct on the panel. |
| kworb_act_check | utils/kworb.py (confirm_act, #4170); call sites KworbClient.artist_standing, youtube_artist_standing, standings, resolve_chart_row | source (streams | listeners | youtube | chart), query (the request credit, truncated to 120), reason (picked | none | judge_failed | all_parts_ranked), count (matched names), + folded detail{answer (the kept name, or “”)} — ONE act check (a classify cache hit emits too; the real model calls are the claude_api rows with purpose=kworb_act_check). artist_standing asks once for both its halves. The strict credit match splits an UNRANKED multi-act credit into parts, so a part can equal a different ranked act (“Zezé Di Camargo & Luciano”, a Brazilian duo, took the German rapper Luciano’s streams row). The known ranked set (#2615) cannot keep an unranked duo whole. When a row matched only through a part (fragment_match), Haiku labels the credit (claude.classify, purpose=kworb_act_check on claude_api, cached 14 days per credit): COLLAB keeps the best-ranked part (the old rule, picked); ONE gives no row (none). judge_failed = an error or an unreadable answer: no row (absent), not cached. all_parts_ranked (#4277) = every part of the request is a ranked act on its own (“George Michael & Wham!”), so the answer is COLLAB with no model call; the panel counts it as no_call. No event when no fragment match happens, or when no judge is attached (scripts, tests). Panel kworb_act_check (Media board: picked / none / judge_failed by source, plus failed_pct). No monitor: the dry run on the live pages reached the check 12 times over 577 requests, so a rate is too thin to gate; a refusal drops one rank to absent rather than going dark. Read a rising failed_pct on the panel. |
| credit_party_check | utils/artist_watch.py (judged_first_party, #3450); call site utils/deezer.py artist_picture_match_for_credit | ok (the judge gave a verdict), decision (yes|no|none) — ONE miss-side first-party check. A co-billed credit (“Drake & 21 Savage”) that missed the whole-credit search and the “+” retry asks Haiku whether the name before the first comma, “&” or “with” is the full name of a music act (claude.classify, purpose=credit_first_party on claude_api, cached 7 days per credit in memory; a cache hit emits too). yes = the portrait search runs on that name and keeps only an EXACT row (whole name, _EXACT_MIN_FANS), reported NON-exact so the media gate still judges it; no / none = absent, as before. No event when no candidate exists or no judge is attached (scripts, tests). Panel credit_party_check (Media board). No monitor: a failed check leaves the card as it was before (no art), and the dry run reached the check 72 times on 707 real credits; read a rising none_pct on the panel. |
| market_label_check | utils/markets.py (confirm_runner_up_cut, confirm_event_platform, KalshiClient._apply_sport_checks, #3425 / #3434 / #3428); call sites KalshiClient.get_events_for_series, get_event_markets, refresh_series_index | source (kalshi), purpose (market_runner_up_check | market_platform_check | market_sport_check), reason (runner-up: cut | kept | judge_failed; platform: picked | none | judge_failed; sport: same | changed | judge_failed), category (runner-up only), count (platform only: platforms the rules name), + folded detail: ticker, topic (the competition, on the sport check), answer (sport check: <ladder tag> -> <model tag>, - = no tag). A runner-up judge_failed KEEPS the event (fail open); a platform judge_failed leaves the title as it is; a sport judge_failed keeps the ladder tag. | Panel market_label_check (Sports board). No monitor: about five title checks a week, and a failure keeps today’s behavior for that one event. The sport check is about 560 rows after each deploy, then none; a burst of judge_failed there means the ladder tags stand, which is the behavior before #3428, so it earns no monitor either. |
| claude_api purpose=riaa_recase / purpose=riaa_same_release | claude_client.py (recase_titles, riaa_same_release), called from utils/riaa.py (recase_texts, resolve_title_collisions) | the claude_api fields only; NO new event and no new field | #3438 / #3959 (2026-10-16). riaa_recase is one Haiku call per RIAA board that is about to compose (only its uncached rows; 30-day riaa_recase cache), and fixes letter case only (“Sza” -> “SZA”). riaa_same_release runs only on a title collision on the plaques window (14-day dry run: one group, one call). Both fail OPEN to the old rule: a model error leaves the rule casing or the merged title, so the card still ships. The failure shows as claude_api ok=false for the purpose. A row the recase answers wrongly is dropped silently (only a log line), because the rule output for that row ships instead. Read: ['tootsies'] \| where event=='claude_api' and purpose in ('riaa_recase','riaa_same_release') \| summarize calls=count(), failed=countif(ok==false) by purpose. No panel and no monitor added: about 2 to 5 calls a day, and every failure falls back to the old card text, which is the state before this change. The existing per-purpose LLM panels already list both purposes. |
| <surface>_scored reason=retro_unframed / reason=retro_interval | cogs/music_news.py, cogs/pop_desk.py, cogs/cinema_news.py, cogs/sports_desk.py | guild_id, score (always 0.0), shipped=false, matchup (subject), post_preview, + folded detail: frame (the wire_retrospective frame the source post carried) — a THROWBACK take was composed and then DROPPED because it stated no look-back frame of its own (retrospective.frames_as_past). The case it comes from (#3270): @Genius posted “eight years ago today, 6LACK dropped ‘east atlanta love letter’”, the desk detected the throwback and told the compose to write a look-back, and the take that shipped to X read “East Atlanta Love Letter debuted at No. 3 on the Billboard 200 with 77,000 equivalent album units in its first week” — past tense, no time frame, so it reads as this week’s debut for a 2018 album. Nothing else catches the class: the self-gate grades whether the claims are GROUNDED and every word of that line is true, so it scored high and shipped. Read this rate against wire_retrospective on the same surface — that is the denominator (how many throwbacks the desk framed) and this is how many lost the frame on the way out. Expect it NEAR ZERO: the note that names the frame was measured on the production compose model with the real inputs at 12/12 framed, against 4/12 for the note it replaced. A sustained rate means the note stopped landing (check whether a block below it in the context turned present-tense — the X CROWD READ block, which says “what people on X are saying RIGHT NOW”, is what pulled the old note off in the dry run), and a rate at or near 100% on one surface means its compose never sees the note at all. reason=retro_interval is its twin on music_news: the take DID frame itself, but with an era the wire never gave it (“back in 1999” off an eight-year-old album), so the numbers inside its look-back clause are not the ones the source fixes (retrospective_values); detail.numbers carries the invented ones. Read it as a fabrication rate, not a framing rate — the self-gate cannot catch it, because the claim is true and only the date is wrong. Its FIRST live hit was a false positive, and the rate is how it was found (2026-09-15). One drop, on a @Genius “today in 2007” wire about “Crank That (Soulja Boy)”: the take quoted the source’s own year back, and the gate called 2007 an invented era because retrospective_values derived the interval 19 from a year frame and never listed the year itself. Read a retro_interval drop against the wire_retrospective FRAME on the same surface — a drop whose detail.numbers is a four-digit year on a today_in / this_day_in source is the take agreeing with its wire, not inventing an era. On music_news the drop marks the post SEEN, so a hit costs that anniversary; that is the trade the WAP re-roll bought (a re-roll pays verify + compose + score again for a phrasing lottery), and the lever that keeps anniversaries in the lane is the note, not a retry. Panel: desk_guard_drops (tootsies-desks), added with this reason — it graphs every deterministic refusal that rides the LANE’s own phase rather than shape_reject, so ungrounded_number and career_ordinal land on it too; before it, those reasons were documented here and graphed nowhere. NO monitor, the same call desk_shape_reject makes: a guard firing is the guard WORKING, and there is no calibrated rate yet that separates “the note is landing” from “a prompt regressed” — the panel builds that baseline first |
| music_news_scored reason=album_unmarked | cogs/music_news.py | guild_id, phase (milestone), score (always 0.0), shipped=false, matchup (subject), post_preview | #album-streams (2026-10-03). An ALBUM stream total (stream_shape=album from the classifier, accepted by album_stream_story) must say “album” in its take, else it reads as a song (the live Jay-Z “Blueprint” 600M post). mark_album inserts the word after the title; this reason fires only when the take names no title to attach it to. Marked seen. A steady low rate is normal; a spike means the compose stopped naming the title. |
| music_news_scored reason=unsupported_rank | cogs/music_news.py | guild_id, phase (story kind), score (always 0.0), shipped=false, matchup (subject), post_preview, + folded detail: claims (the flagged phrases) | #3838 (2026-10-05). On a story whose post text is ground truth (a number authority on X, a feed item, a catalog post), the take states an ordinal, a superlative’s rank or an all-time scope that the post and the desk’s own readings do not support (output_checks.unsupported_rank_claims). The web background and the classifier’s claim are NOT its source. Live cases: “her second album” on Adele’s “25”, “the most-streamed metal album ever” off “the top two”. Marked seen. Panel desk_guard_drops. A rate above a few a day means the compose is adding clauses again; check the SOURCE POST block still reaches it. |
| music_news_scored reason=unsupported_title | cogs/music_news.py | guild_id, phase, score (always 0.0), shipped=false, matchup, post_preview, + folded detail: claims (the titles) | #3838 (2026-10-05). The twin of unsupported_rank: on a source-post story the take quotes a title the post and the desk’s own readings never name (mn.ungrounded_quoted_titles). Live case: Wiz Khalifa’s catalog total gained “built mostly on the back of "See You Again"”. Marked seen. Panel desk_guard_drops. Since #4583 (#4553) the word match is the fast path: each title it misses goes to the title judge (purpose title_in_post), and a title the judge finds in the source emits music_news_recover detail.via=title_judge_take and does not drop the take. A judge error drops, as before. Measured: 1 drop in the 30 days to 2026-10-24. |
| music_news_scored reason=wrong_move | cogs/music_news.py | guild_id, phase, score (always 0.0), shipped=false, matchup, post_preview, + folded detail: numbers (the move sizes) | #3853 (2026-10-05). The take states a move size (“up 21 spots”) that equals no move the post states (mn.wrong_move_sizes: “#1(+20)”, “vaults 21-1”, “up N spots”). A post with no move checks nothing. The classifier wrote “(up 21 spots)” off “vaults 21-1” 10 of 10 runs before its example was reworded, and mn.strip_wrong_moves now cuts that clause from the claim too. Marked seen. Panel desk_guard_drops. A non-zero rate means the compose computed a move; read the preview. Since #3864 a size written in words (“twenty-one spots”) is read too, and with no move in the post a WORD size is flagged, because the digit gate cannot hold it; wire_move_sizes also reads “from #9 to #8”. |
| music_news_scored reason=chart_frame | cogs/music_news.py | guild_id, phase, score (always 0.0), shipped=false, matchup, post_preview, + folded detail: claims (the words) | #3954 (2026-10-09). The take names a chart period its ground does not (“daily” over a “Weekly” claim), or adds a word that says the position did not hold (“briefly”, “temporarily”) that the ground does not state (mn.misstated_chart_frame). The ground is the source post and the desk’s own evidence (the classifier claim only when there is no post text), never the web research: on the live case the research proposed the wrong sentence. Only a period word that names a chart counts. A ground with no period checks no period. Marked seen. Panel desk_guard_drops. A non-zero rate means the compose took a frame from the background; read the preview. |
| music_news_scored reason=confidence_hedge / reason=wrong_credit | cogs/music_news.py | guild_id, phase, score (always 0.0), shipped=false, matchup, post_preview | #3856 (2026-10-05). The take wrote a confidence disclaimer (“can’t verify”, “remains unverified”, “reportedly”; output_checks.confidence_hedge_hits) or named a platform or a market as the counter (“per YouTube’s own count”, “per the market”; self_count_credit_hits). The confidence signal is the tag and the card’s source pill (owner steer). The prompt causes were cut first (0/15 hedged takes on the ablation set), so this is a backstop. Marked seen. Panel desk_guard_drops. Since #3864 the adjective use on another fact (“an unconfirmed release date”, “the reportedly shelved album”: a determiner before the word, an allowlisted non-figure noun in the next two words, and no digit, number word or figure noun in the next three) is not a hit. Any other wording still counts, so the exemption fails closed. Since 2026-10-06 reason=wrong_credit also covers a credit to an UNNAMED source (anonymous_credit_hits: “third-party”, “a/one/some tracker”, “fan-run count”, “unofficial tally”). The shipped line was “A third-party tracker puts Drake at 1.7 billion Spotify streams across all credits in September.” A named tracker (“per ChartMasters”) and the verb “tracking for” still pass. |
| music_news_scored reason=later_week | cogs/music_news.py | guild_id, phase, score (always 0.0), shipped=false, matchup, post_preview | #3860 (2026-10-05). The post is about a later week of a title already out (mn.post_states_later_week: re-peak, return to #N, re-entry, a numbered week past the first), and the take calls a figure “first-week” that the post does not attach to a first week (mn.later_week_first_week_claim). The classify parse already moves such a story off first_week, so this is a backstop. talkofthecharts’ HABIBTI re-peak shipped “65K first-week units” 3 of 3 runs before the fix. Marked seen. Panel desk_guard_drops. Expect near zero. |
| music_news_scored + detail.unconfirmed_penalty | cogs/music_news.py | guild_id, phase, score (the JUDGE’s own score, unchanged), shipped (read off the gated score), matchup, post_preview, + folded detail: unconfirmed_penalty (0.25), gated_score (score minus the penalty) | Owner steer 2026-10-09, after the Madonna “260 million equivalent album sales” post: @chartmastersorg was the only source, the verify judge found no other, the take scored 0.82 and shipped, and the owner deleted it. A number is UNCONFIRMED when the judge did not confirm it and no live reading settles it (settled is False). The ship floor then reads score - UNCONFIRMED_SCORE_PENALTY, so a take under 0.85 does not ship. A release story (no number) takes no penalty. A penalty-only drop (raw score at or over 0.6) is not marked seen, so a later slot can ship it once confirmed. score keeps the judge’s value so the ops monitor’s quality MEAN still measures the writing; its below-floor count (surface_dark / lane_dark / low_quality) reads detail.gated_score when present, so a lane the penalty silences still reads dark. Panel outcomes: shipped, penalty_drop, below_floor. Measured over the 14 days before: 79 of 372 posts were unconfirmed, about 16 of them under 0.85. Panel newsroom_unconfirmed. No monitor: a policy cost, read weekly. |
| music_news_recover via=live_leg / music_news_filtered reason=market_unread | cogs/music_news.py | guild_id, ok=true, phase (market), matchup, + folded detail: via, figure (the live percent), was (the wire figure), title (the Kalshi/Polymarket event) / guild_id, phase, reason, matchup, source, + folded detail: claim, venue, matched | 2026-10-06, the Drake / Coachella post. A wire said Drake had “a better than 50% chance” to headline Coachella 2027 and the newsroom relayed it as “50%+” while Kalshi had the leg at 62.6%. A single-outcome market story that names an exchange now has its odds replaced by the LIVE leg (_live_market_odds: Kalshi via our open-event index + kalshi_price.market_yes_price, Polymarket via its public search; mn.pick_market_leg picks the event and leg). No leg, or a claim with a second number the price cannot restate, drops as market_unread. A live leg under mn.MARKET_STORY_MIN_PROB (25%) drops as market_longshot (detail: claim, venue, figure) and marks the post seen (2026-10-10, the Tate McRae “7% chance to debut at No. 1” post the owner did not need). Dashboard: newsroom_market_reads. No monitor: a few market stories a week is too few for a 15-minute window; read the panel ratio instead. |
| music_news_recover via=title_judge / music_news_filtered reason=title_ungrounded detail.rescue | cogs/music_news.py | guild_id, ok=true, phase (story kind), matchup, + folded detail: via / the drop row gains folded detail.rescue = none | unquoted | error | #3410 wave 19 (2026-10-18). The title gate’s word match (mn.title_grounded_in_wire) is the fast path. A miss asks Haiku for the post’s own words that name the title (claude_api purpose title_in_post), and mn.quote_grounds_title keeps the story only when the quote is in the post and shares a content word with the title. Measured on 47 real drops (2026-09-27..10-18, 3 runs): 11 kept that the post does name, 0 memory titles kept (GUTS, L.A.M.B., DtMF), 1 event description kept on 2 of 3 runs (“GOLF” for Tyler’s El Clásico jersey). rescue=error is the silent degradation: the judge is down and the gate drops as it did before #3410 wave 19, which is the safe direction. Dashboard: newsroom_title_judge. No monitor: about 6 checks a day, and every failure falls back to the old drop. |
| music_news_recover via=rank_position | cogs/music_news.py | guild_id, ok (a position heroes), phase (story kind), matchup, source, + folded detail: via, was (the band, “top 10” / “Top 2”), figure (“#4”, the best position the subject holds, or “”), reason = story | search | unstated | ungrounded | none | 2026-10-18, the Taylor Swift “Top 2” card. Owner steer: “stop doing Top N, and just always name the position like #N”. A card figure that is only a band, after the live chart read, goes to Haiku (ClaudeClient.name_chart_positions, claude_api purpose music_news_rank_position) with the post and the verify research. When neither names a position, one claim-scoped Perplexity search runs (pplx_music_news_rank) and Haiku reads it. The answer must sit inside the band and be written as a position in that text (mn.position_grounded), else the figure goes empty and the card falls to headline mode. story and search are the check working. unstated is a band with no position anywhere, the expected miss. none is the silent degradation: Haiku gave no verdict, and the card dropped its figure. ungrounded means the model named a number the text does not carry. Dashboard: newsroom_rank_position. No monitor: about 1 to 2 a day (30 days to 2026-10-18 had 45 band figures), and every failure drops the figure, which is the safe direction. |
| music_news_recover via=ranked_stat | cogs/music_news.py | guild_id, ok (the stat was found, or none was missing), phase (story kind), matchup, source, + folded detail: via, figure (“1,113 days” or “”), reason = story | search | not_ranked | unstated | ungrounded | none | 2026-10-24, the JENNIE “2nd” card (“second longest-running active streak among female artists on the Global Spotify chart”, no day count; the verify note held “1,113 days”). Owner steer: “Always say the actual stat. Never top N, first, second, third.” A claim with a ranking word (mn.claim_may_rank) goes to Haiku (ClaudeClient.name_ranked_stat, claude_api purpose music_news_ranked_stat) with the post, the verify research, the X context and the verify note. When none states the stat, one claim-scoped Perplexity search runs (pplx_music_news_ranked_stat, last week only) and Haiku reads it. The stat’s number must be written in the text it came from, with its unit word right after it (mn.stat_grounded; the unit keeps “Hot 100” from matching “101 weeks”, Codex review on #4755). A found stat is added to the claim (mn.claim_with_stat) and heroes the card. unstated and ungrounded drop the story (music_news_filtered reason=ranked_no_stat); a take that leaves the stat out drops too (music_news_scored reason=ranked_stat_missing). none is the silent fault: the claim ships as written. Dashboard: newsroom_ranked_stat. No monitor: a few a day (16 of 400 real lines in the 7 days to 2026-10-24 were a ranking with no number), and every failure but none drops the post, the safe side. |
| music_news_recover via=artist_rescue / music_news_filtered reason=unknown_artist detail.rescue | cogs/music_news.py | guild_id, ok=true, phase (story kind), matchup, + folded detail: via, artist (the tracked act), was (the cut credit) / the cut row gains folded detail.rescue = none | no_candidates | error | #3439 (2026-10-15). A story the tier gate is about to cut as an unknown artist asks a cached Haiku closed-set check (aw.rescue_unknown, claude_api purpose watched_act_match) against the nearest tracked names. rescued keeps the story. rescue=error is the silent degradation (no client, a model error, an answer outside the list): a run of it means the judge is down and the gate is back to its old cuts, which is the safe direction. Dashboard: newsroom_artist_rescue. No monitor: about 5 checks a day is too few for a window, and every failure falls back to the old cut. |
| luminate_subject | utils/music_markets.py | ok, reason (changed / agreed / error), ticker (the Kalshi event), topic (the subject after the check), + folded detail: was, metric, was_metric | #3436 (2026-10-16). The Kalshi music sweep reads each event’s act, release and metric with the cue rules (luminate.metric_cues + _SUBJECT_RE). An event they cannot settle (no metric cue or two different ones, no leg that parses, or two acts in one comparison) asks one cached Haiku call (claude_api purpose luminate_subject). Every value is checked against the text; a comparison of two acts credits no act. error = no client or a model error; the cue result stands, which is the pre-#3436 behaviour. One row per real model call (answers cached 7 days, errors 10 minutes). Dry run 2026-10-16 on 121 live events: 50 reached the model, 14 changed. Panel luminate_subject_check. No monitor: an error is never worse than before. |
| music_news_recover via=chart_scope_model / music_news_filtered reason=chart_cut detail.via / music_news_recover via=forecast_wording detail.read | cogs/music_news.py | guild_id, ok=true, phase (chart / no_chart), matchup, + folded detail: via, family / the cut row gains folded detail.via = model | rule / the forecast row gains folded detail.read = model | rule | CB-01 (sub-epic #4557, 2026-10-18). One cached Haiku read per story (claude_api purpose chart_story_read) names the chart family, forecast, later week, edition and what the title is (song / album / tour); a second label (take_marks_album) says whether an album take already says album. A chart NAME in either scope list still decides with no read; the model decides only the residue that names neither. chart_scope_model = a story the model kept that the rule cuts as unnamed (measured: 10 of the 11 real kept charts in 30 days of unnamed cuts, 2 runs). detail.via=rule on a residue cut, or read=rule on a forecast, is the silent degradation: the read gave no answer and the old word lists decided, which is the pre-CB-01 behavior. Dashboard: newsroom_story_read. No monitor: about 60 reads a day, and every failure falls back to the old rule, not to silence. |
| music_news_filtered reason=ladder_rung detail.via | cogs/music_news.py | the drop row gains folded detail.via = model | rule | CB-08 (#4557, 2026-10-18). Haiku (claude_api purpose market_rung, ClaudeClient.market_claim_is_rung) decides whether a market claim quotes one rung of a threshold ladder; the regex quotes_ladder_rung is the fallback. Measured: the regex read a year as a threshold (“the top US artist in 2026”, 7 drops in 30 days). via=rule is the silent degradation: the label gave no answer. Dashboard: newsroom_market_reads (ladder_rung:model / ladder_rung:rule). No monitor: about 1 market story a day. |
| music_news_recover via=tracker_lag | cogs/music_news.py | guild_id, ok=true, phase (milestone), matchup, source (the authority handle), + folded detail: via, note (the judge’s note) | #3838 (2026-10-05). The verify judge refuted a number authority’s milestone total, and the note’s figure is under the post’s rung by less than 2% or is dated before the post (mn.refute_is_tracker_lag). The verdict becomes UNCONFIRMED, the story is relayed on source trust, and the lagging research is cleared before compose. Live case: Fall Out Boy’s “passed 2 billion” refuted by kworb 1,975,713,319 while kworb’s own page read 2,002,211,599. Panel newsroom_refutes (beside stale_refute). |
| music_news_recover via=over_rung / via=wrong_match | cogs/music_news.py | guild_id, ok=true, phase (milestone), matchup, source (the authority handle), + folded detail: via, note (the judge’s note) | #3855 (2026-10-15). The verify judge refuted a number authority’s milestone total with a figure that is not evidence against it. over_rung: a note total sits above the post’s rung by at most 10% and no note total sits under it (mn.refute_is_over_rung); live case: Nelly Furtado’s “8 billion” refuted by “Songstats measures 8.6B”. wrong_match: every note total is under a tenth of the rung, so the research read another item (mn.refute_is_wrong_match); live case: Sean Paul’s “Temperature” at 1.5B refuted by Songstats’ 963.1K for another recording. A claim that also states a small count (“3 songs over 3B”) is read by neither rule. Same outcome as tracker_lag, and the Songstats block is also withheld from the compose and the gate source. On the 30-day replay these recover 17 of the 60 authority milestone refutes. Panel newsroom_refutes. |
| music_news_recover via=research_dispute | cogs/music_news.py | guild_id, ok=true, phase (the story kind), matchup, source (the wire handle), + folded detail: via | #3957 (2026-10-15). A culture-wire story the verify judge CONFIRMED (from X posts) whose Perplexity research DISPUTES the claim. Since #3437 the verify judge answers this in the same call (web_disputes in its JSON, no extra call); perplexity.research_disputes is the fallback when the judge gives no flag or did not see the research (an artist pull). The regex reads the answer’s opening verdict (“partly accurate but not fully verifiable”, “not currently verifiable”, “conflates …”), never the body, which nearly always says “exact current figure: not verifiable”. The research then leaves the compose AND the digit/ordinal gate source; the judge still read it. The case: @rap’s “Drake rises to #1 on Spotify’s Weekly Top Artists Global chart” (2026-10-09) shipped as “Drake briefly rose to No. 1 on Spotify’s global daily artist chart” off research that mixed it up with a daily #1. Measured on the production model, n=6 per arm: with the research 6/6 takes carried a research-only fact and 0/6 shipped; withheld, 0/6 and 6/6; a softer header instead, 5/6 and 1/6. On 12 real confirmed stories (2026-10-15) the test flagged 3, and all 3 openings did dispute the claim. No monitor: it is a cut toward absent, and a burst means wires the web cannot confirm, not a fault. Not on a dashboard panel yet (the newsroom_refutes panel lists its via values by name; follow-up #4130). |
| music_news_recover via=forecast_wording | cogs/music_news.py | guild_id, ok=true, phase (chart), matchup, + folded detail: via, flag (the classifier’s is_projection read, False) | #3874 (2026-10-05). The classifier left is_projection empty on a post that states a forecast (“‘The Life of a Showgirl’ to return to #1 … with 172K units”, 3 of 4 runs), and last week’s page (#21) settled the claim. mn.post_states_forecast reads strict forecast wording (“projected”, “set to”, “on pace”, “will” + a chart verb, the headline infinitive “X to return to #1”) in a sentence that states a position and names the subject’s title, and makes the story a projection. Each row is one classifier miss the wording caught. A steady low rate is normal; a rising rate means the classifier’s projection call is drifting. |
| music_news_recover via=riaa_level | cogs/music_news.py | guild_id, ok (a level was found), phase (cert), matchup, reason on a miss (feed_incomplete = the newest-awards walk was short or capped; not_in_feed = the 14-day feed does not hold the title; feed_level = it holds the title only below 2x, or at two levels), + folded detail: via, level (RIAA’s badge level, null on a miss), format (ALBUM / SINGLE / "", the format the claim names) | 2026-10-10, owner report on the Ella Langley “Dandelion” card. The wire wrote “multi-Platinum” with no count, and the card heroed that word. RIAA has no award by that name. _riaa_level_for reads the level RIAA holds for the title off a complete, fresh (uncached) walk of RIAA’s newest-awards feed, 14 days back (no cache is read) and the card heroes “2x Platinum”. It also sets cert_id (2xplatinum|riaa), so the story dedups against the RIAA lane’s own post of the same award. ok=false keeps the wire’s word: no row, rows that disagree on the level, or another authority. No new panel: it rides the existing music_news_recover panel. |
| music_news_filtered reason=chart_exit | cogs/music_news.py | guild_id, phase (chart), matchup (subject), source (wire handle), + folded detail: via (classifier / regex) | #3565 (2026-10-05). A chart story reports a title LEAVING the chart (mn.states_chart_exit: “exits”, “falls off the Hot 100”; not a top-N tier, a negation, a return or a claim with a position; the post is read only for a sentence that names the subject’s title). Cut before verify and compose and marked seen, the newsroom twin of the desk’s MUSIC_DESK_EXITS cut (owner steer 2026-09-21). MUSIC_NEWS_EXITS=1 restores the relay. Since #4201 the classifier’s chart_event decides first: exit cuts inside the same bound (exit wording or “departs”, in the claim or a sentence that names the title), where a “from #N” or a peak position does not block; detail.via=classifier. Any other event is no exit. via=regex is a reply with no event. Since #4583 (#4553) detail.title says what tied the exit clause to the subject’s title: word (the word match) or judge (the title judge, purpose title_in_post, asked only on an exit clause the word match misses; a judge error is no exit). Measured: 0 cuts in the 30 days to 2026-10-24. |
| music_news_filtered reason=position_only | cogs/music_news.py | guild_id, phase (chart), matchup (subject), source (wire handle), + folded detail: via (the classifier’s chart_event, hold / none) | 2026-10-23 (owner steer “only rise”, the newsroom twin of the desk’s music_desk_trim reason=fall). A chart story that says only where a title IS, below #1 (mn.position_only_claim): “Dandy Hats’ ‘El Rabbanito’ is #9 on Billboard’s Hot Latin Songs chart”. The classifier’s chart_event decides whether the wire names a move: only hold (same position as last week, which includes “an Nth week at #3”) and none (a position with no week-on-week verb) are cut; debut / reentry / move / exit and an EMPTY event are never cut. The position is a closed-format regex (“No. N”, “#N”); an earlier draft also used open move-word and record-word lists, which broke the owner rule that an open set of words goes to a model (tests/test_classifier_registry.py), so the model’s event replaced them. Since the same day a HELD #1 is cut too (“Housewife is No. 1 … its third day on top”; owner: level posts “aren’t relevant and often redundant noise”, and the alert cog posts every NEW #1 itself). A none #1 is kept, because the wire may be the first report of a new #1 or a record (“hits No. 1 … for the first time ever”). A “top N” presence stat and a projection (is_projection) are kept. Cut before verify and compose and marked seen. Measured on the 7 days before the cut: 20 of 208 shipped newsroom chart posts (about 3 a day), mostly Mediabase radio positions and Spotify daily positions. Since 2026-10-27 a FORECAST hold is cut here too, read off the classifier’s chart_direction (level + is_projection, mn.direction_cut; detail.via=direction): “Housewife is aiming to spend a second consecutive week at #2, per early forecasts”. A settled hold stays with position_only_claim, because on two runs over 100 real wire posts the model read some career records (“ties … as her highest, both #4”) as level on one run in two. No monitor: a zero count is a quiet wire, not a failure, and the newsroom going dark is covered by surface_dark. |
| music_news_filtered reason=fall | cogs/music_news.py | guild_id, phase (chart), matchup (subject), source (wire handle), + folded detail: via (direction), event (the classifier’s chart_event) | 2026-10-27 (owner steer “only rise”). A chart story the classifier’s chart_direction (Haiku, classify_music_news) reads as a FALL, settled or forecast (mn.direction_cut). Before it, position_only_claim read only chart_event, and a move is a climb or a fall, so a fall (“Billboard Hot 100: #7(-3) Been By Now”) reached verify and compose and only the wording checker stopped it after compose; a forecast was never cut, and “drop dead is expected to fall below the top 25 and go recurrent” shipped on 2026-10-26. Measured on 100 real wire posts (2026-10-27, two runs): 7 of 7 falls read down on both runs, and no other post read down. A debut, a re-entry or an exit is never cut here (an exit is the exit gate’s, MUSIC_NEWS_EXITS); an EMPTY direction cuts nothing. Cut before verify and compose and marked seen. No monitor, same reason as position_only: a zero count is a quiet wire. |
| music_news_recover detail.via=settled_move | cogs/music_news.py | guild_id, phase, matchup, ok (a move kept), + folded detail: move (climb / debut / none) | 2026-10-27. A live chart read rewrote an UP story’s claim (mn.settled_claim), which dropped the wire’s move: “WTF GOIN enters the Top 5 on today’s US Rhythmic Radio rolling chart” shipped as “sits at #4”, and a rise read as a hold. Haiku (purpose settled_chart_move, ClaudeClient.settled_chart_move) reads whether the live position still fits the wire’s move; a kept move rides back as a word (“climbs to #4 on …”), never as the wire’s number. The row facts overrule it: a weekly row whose last week was no lower is not a climb, and a title the chart memory has as established is not a debut. Measured on 92 real wire/live pairs from the 7 days before (three runs): 35 of 37 real moves kept, and 0 moves claimed on the 24 pairs where the live position contradicts the wire (“rises to No. 6” vs live #17, “enters the Top 20” vs live #21). The Haiku call shows on claude_api purpose=settled_chart_move. No monitor: ok=false is the safe direction (the position alone, the old behavior). |
| music_news_filtered reason=itunes_depth | cogs/music_news.py | guild_id, phase (chart), matchup (subject), source (wire handle), + folded detail: pos | 2026-10-23 (owner steer “lets cut itunes to top 5”, “for both”; Codex on #4679). An iTunes chart story (chart_key in mn.ITUNES_CHART_KEYS) whose claim states a position below x_gate.ITUNES_TOP_N (#5), debut and move alike, the newsroom twin of music_desk_trim reason=itunes_depth. A projection is not cut. Cut before verify and compose and marked seen. |
| music_news_recover via=chart_key_scope | cogs/music_news.py | guild_id, ok=true, phase (chart / no_chart), matchup, + folded detail: via, family (the kept platform family), key (the classifier’s chart_key) | #3448 (2026-10-16). A chart story that neither KEPT_PATTERNS nor CUT_PATTERNS could name (chart_cut family=unnamed) is kept when the classifier’s chart_key names a Spotify daily chart and “spotify” is in the text. Live misses it fixes: “opens at No. 46 on the Spotify Global chart”. A Billboard key on an unnamed story stays cut: the regex reads every Billboard name the post states, so that key is the model’s memory. Shown as kept:<family> on the desk_chart_scope panel. |
| music_news_recover via=chart_event_not_exit | cogs/music_news.py | guild_id, ok=true, phase (chart), matchup, + folded detail: via, flag (the event) | #4201 (2026-10-16). The exit regex reads an exit and the classifier’s chart_event says another event, so the story is NOT cut. Live case: “Rosé and Bruno Mars Lead ARIA Charts Shake-Up After Festive Exit”, a #1 story the regex cut. A steady count of real exits here means the field is keeping exits; read the matchups. |
| music_news_recover via=event_pick / via=count_collision | cogs/music_news.py | ok (event_pick: a leg was picked; count_collision: the model gave a verdict), phase (market / chart), matchup, + folded detail: via, count (candidate events / colliding listings), title (the picked event, event_pick), verdict (event_pick, CB-05: event / none / floor = the year-role floor vetoed the pick / error = no verdict, the code rule decided), agree (event_pick: the model’s pick equals the code rule’s), chart + reason (same / different, count_collision) | #4202 (2026-10-16). event_pick: since CB-05 (#4557) on every live read, not only a tie: Haiku (claude_api purpose market_event_pick) picks one of the events that list the subject (mn.subject_market_legs) or NONE, and mn.market_role_conflict (year / role) vetoes a pick as a safety floor. Regression query: ['tootsies'] | where event == 'music_news_recover' and tostring(parse_json(detail)['via']) == 'event_pick' | summarize count() by v=tostring(parse_json(detail)['verdict']), a=tostring(parse_json(detail)['agree']); agree=false rows are where the model changed the outcome: read their titles. count_collision: two rows on one chart share a title but not a lead credit (#3664), and Haiku (purpose count_same_record) says whether they are one record; only a different verdict splits the count. Both are rare: 0 of each in the dry run. |
| wire_retrospective reason=cut | utils/retrospective.py (cut_throwback; called first in _compose_one of cogs/pop_desk.py, cogs/cinema_news.py, cogs/sports_desk.py) | surface, guild_id, matchup (subject), source (wire handle), reason (cut), + folded detail: frame | Owner steer 2026-10-20: a throwback ships on no wire desk. The regex frames decide first; Haiku (prime_throwback, wire_throwback_check) reads a post the regex misses. The desk drops the story before the Grok news-age read, and its caller tries the next story. Dry run 2026-10-20, live Haiku, 250 real posts from 14 wire accounts: 12 cut (7 regex, 5 Haiku only), 0 current posts cut. The old <surface>_scored reason=retro_unframed guard on these three desks was removed with the reframe path. Panel: wire_throwback_outcome. No monitor: a miss ships one post. |
| music_news_filtered reason=milestone_watch | cogs/music_news.py (_milestone_watch_cut) | guild_id, phase (milestone), matchup (subject), source (wire handle), + folded detail: tier (old / no_date / not_watched) | Owner steer 2026-10-21: a milestone posts for a WATCHED act (the kworb top 100), or for another act only when its title is from the last two years (recency.age_verdict, the shared release-date resolver). A bare-artist subject by an unwatched act (a career total) has no title to date and is cut as not_watched. Cut before the verify; no_date is not stamped seen. Fail-open when the watch holds fewer than 50 names; off with MUSIC_DESK_RECENT_CUT=0. Dry run 2026-10-21 on the 419 milestones posted in the 30 days before: 266 watched, 17 recent, 136 cut (97 old, 37 career totals, 2 undated). Panel: release_age_cut (lane milestone). No monitor: a miss ships one post. |
| music_news_filtered reason=past_cohort | cogs/music_news.py (_compose_one, utils.music_news.past_cohort_rank) | guild_id, phase (the story kind), matchup (subject), source (wire handle), + folded detail: cohort (1972 / 1980s) | Owner steer 2026-10-21: a story whose hero is a rank on a list limited to a past release year or decade (“Harvest is now No. 6 among 1972’s most-streamed albums”) is cut first in _compose_one, before the CB-01 read, the verify and the compose, and marked seen. Lists for this year and last year pass. A milestone that only carries such a rank (its hero is the stream count) passes. Panel: release_age_cut (lane rank_list). No monitor: a miss ships one post. |
| music_news_filtered reason=throwback | cogs/music_news.py (cuts_throwback) | guild_id, phase (the story kind), matchup (subject), source (wire handle), + folded detail: frame (the retrospective_frame slug, model when Haiku read it) | The music newsroom cuts every throwback. A throwback is dropped before the verify and the compose, and marked seen. History: 2026-10-19 cut only the chart / first_week kinds plus a Haiku chart-position read on the other kinds (reason=chart_throwback, detail.via, claude_api purpose=throwback_chart). On 2026-10-20 a cert anniversary shipped to X (“23 years ago today, Amy Winehouse’s ‘Frank’ was certified 4x Platinum”) and the owner asked for all throwbacks to go, so the kind list and the Haiku read were removed. No wire desk ships a throwback any more: pop_desk, cinema_news and sports_desk cut theirs as wire_retrospective reason=cut (row above). A wire_retrospective row with no reason is the older reframe path. Panel: wire_throwback_outcome. No monitor: a miss ships one post framed as past. |
| music_news_filtered reason=album_shape_conflict | cogs/music_news.py | guild_id, phase (milestone), matchup (subject), source (wire handle), + folded detail: claim | #album-streams (2026-10-03, #3811). The classifier said stream_shape=album but the story is not one album: the post calls the title itself a song (“Lover, a track from the album Lover”), or the subject is a bare artist. The classifier writes “album” into an album claim, so the claim is not trusted either; the story is dropped and marked seen. A steady low rate is normal; a spike means the classifier’s album/track call drifted. |
| music_news_recover via=qualifier_cut / via=qualifier_unchecked, music_news_filtered reason=qualifier_wrong, music_news_scored reason=cut_qualifier | cogs/music_news.py | guild_id, ok, phase (story kind), matchup (subject), source (wire handle), + folded detail: via, cut (the removed phrases), note (the judge’s reason); the scored row carries detail.cut (the qualifier words) | The QUALIFIER CHECK (2026-10-17, the Tate McRae “first time ever” post). A claim with a qualifier gets its own research search (qualifier_check.qualifier_research, event pplx_music_news_qualifier, no recency window; #4486) and is asked of check_claim_qualifier (Sonnet, claude_api purpose=music_news_qualifier). A Spotify Global “first” claim also gets the kworb Global history (chart_fetch query=global_daily_totals; #4491). qualifier_cut: a wrong qualifier was cut and the rest shipped. qualifier_wrong: the qualifier is wrong and the cut was not clean, so the story is dropped and marked seen. cut_qualifier: the take stated a cut qualifier word again, so it is dropped and marked seen; a pre-judge score=0.0 stamp in NON_SCORE_REASONS. qualifier_unchecked (ok=false): the judge gave no verdict and the claim shipped as written. A run of qualifier_unchecked is the fault to look for; the other outcomes are the check working. Panel newsroom_qualifier; no monitor (a few a day). |
| owner_ban | utils/ship_guard.py (owner_ban_label, CB-03 #4557) | reason (the fence: attribution_tag / confidence_hedge / cross_source_comparison / stream_unit_comparison / row_arithmetic / prior_position), ok, shipped, post_preview, + folded detail: label | One row per owner-ban fence hit. The regex in utils/output_checks.py finds the words and a cached Haiku read (claude_api purpose=owner_ban) says what they do in the line. shipped=true is a CLEAR: the line passes that fence and no drop event follows, so this is the one place a clear shows. ok=false means no model answer; the regex verdict stands and the line drops (these are owner bans, so they fail CLOSED). The drops themselves keep their <surface>_scored reason=<fence> rows, now with detail.via (model / rule). Callers: the music desk (_bans_cleared before _unshippable, and detag_credit), music_alert (settled-alert credit), music_news (hedge / credit, prior position through ship_guard.drop_prior_position), market_drop (cross source), sports_boards and cinema_desk (row arithmetic). Expect about 1 row a day (19 fence drops in the 30 days to 2026-10-18). Panel owner_ban (Desks board). No monitor: a model failure keeps every drop, so it cannot ship a banned line; the risk is a wrong CLEAR, and at about one read a day no rate is calibrated. Read the panel’s shipped=true rows. |
| qualifier_shape | claude_client.py (qualifier_shape, #4522) | ok, reason (model / error / unreadable), count (time phrases listed), duration_ms | One row per claim that reaches check_claim_qualifier. A cached Haiku read (claude_api purpose=qualifier_shape) says which #4512 shape rules the judge gets: the time phrase that limits a qualifier, “joins X and Y as the only”, a maximum “most”. ok=false: no model answer, so the regex gates decided (the rule before #4522). reason=error covers every raise in the read (_call, the cache, .strip(), json.loads; #4537), so a fault in the read gives the regex gates, never no rule. The fallback share is the fault to look for. Panel qualifier_shape; monitor: the ops-monitor health line qualifier_shape. |
| qualifier_present | claude_client.py (claim_has_qualifier, #4522) | ok, reason (yes / no / no_verdict), count (candidate words), duration_ms | One row per claim the qualifier word list (qualifier_trim.qualifier_candidates) fires on; a claim with no candidate makes no read. A cached Haiku read (claude_api purpose=qualifier_present) says whether a candidate is a real qualifier. no skips the qualifier research and judge (a false hit such as “58-63k first week”). ok=false: no verdict, so the word list’s answer stood and the judge ran (the rule before #4522). It raises nothing, so the fallback share is the fault to look for. Panel qualifier_present; monitor: the ops-monitor health line qualifier_present. |
| qualifier_check | utils/qualifier_check.py (any lane that repeats a source; epic #4412) | surface (the lane), guild_id, ok, reason (kept / cut / no_clean_cut / unchecked), matchup (subject), + folded detail: cut (the removed phrases), note (the judge’s reason, or no_research) | One row per source claim with a qualifier that a lane checked with check_source_qualifier (2026-10-17). cut: a wrong qualifier was cut and the rest ships. no_clean_cut: the lane drops the story. unchecked (ok=false): no research or no verdict, the claim ships as written; a run of these is the fault to look for. A claim with no qualifier emits nothing. The take-drop side is each lane’s <surface>_scored reason=cut_qualifier. The newsroom keeps its own music_news_recover via=qualifier_* events. Surfaces (2026-10-18): sports_desk, pop_desk, cinema_news, cinema_desk, music (the music drop), music_desk, market_alert, music_alert, x_reply_draft (the X reply and banger lanes, trigger=reply|banger; #4464, #4467), x_mentions (the mention reply lane’s parent post, research purpose x_mentions_qualifier; #4467). Each lane’s judge call is purpose=<surface>_qualifier, and its research is the Perplexity purpose <surface>_qualifier (the music drop uses music_drop_qualifier). market_alert has no *_scored event: its take drop is market_filtered reason=cut_qualifier. The X reply lane has none either: its drops are x_reply_draft reason=qualifier_no_clean_cut and x_reply_draft reason=cut_qualifier. On both alert lanes a news line with no clean cut is market_news_read reason=qualifier_no_clean_cut, and the alert ships price-only. Panel lane_qualifier (rows by lane and outcome, the take drops included, market_filtered as lane market_alert, x_reply_draft reason=cut_qualifier as lane x_reply_draft). Monitor: the ops-monitor health line qualifier_check_<surface> (the share of checks with a verdict), because unchecked fails open and raises nothing. |
| deezer_lookup / deezer_track | utils/deezer.py (_get_json, via every read’s timed_event block) | query (the search term or a artistpic: / albumcover: / chart: / track_ prefixed key, truncated), hit, ok, error, duration_ms — ONE Deezer read. ok told the truth only from 2026-09-15 (#deezer-403). _get_json is fail-open: it catches its own error and returns None, so a dead Deezer left the surrounding block through a clean exit and timed_event stamped ok=true. Measured over the 2026-09-14/15 outage, while Deezer answered 403 to every call, deezer_lookup reported ok=true on 402 of 402 reads. The only trace was an error event marked recoverable, which error triage discounts — the exact fail-open blind spot the integration-health rates exist to cover. _get_json now calls event().fail(...) on a failed read, and on an {"error": ...}-with-HTTP-200 body whose code is NOT 800. Code 800 (DataException, “no data”) is deliberately excluded: Deezer answers both outcomes through the same envelope, and 800 is a HEALTHY request for a row that does not exist (a delisted track id, an artist with no album), verified live against /track/999999999999. Counting it would report an outage every time a pooled clip id went stale — the same ok=False contamination catalog_search had to correct. A read that SUCCEEDS also clears an earlier failure in the same block, because fail() latches and several blocks read more than once (album_cover tries two search terms then the discography rung), so without it a block that missed once and then found the cover reported ok=false while the card shipped correct art. Both kinds are registered in ops_monitor._MISC_INTEGRATIONS under ONE deezer bucket, not split per query: the query field is a raw search string, so a per-query split would make one throwaway bucket per artist, and this failure is total anyway — the host refuses every call or none. Deezer is the repo’s ONLY source of an artist PHOTO and it is fail-open twice over (the client returns None, each caller reads None as “no match”), so a dark Deezer silently empties the portrait rung and the genre boards rather than raising. This rate is the only thing that says so. |
| portrait_sharpness | utils/catalog_art.py (portrait_blurred, #2716; one shared check since #4134), called by utils/entity_image.py (_deezer_photo_bytes), utils/music_markets.py (_portrait_rungs, the music-card ladder, surface music_markets) and cogs/music_desk.py (_fetch_art’s artist photo, surface music_desk) | source (deezer), surface (absent for entity_image), reason (sharp | blurry), score (Laplacian variance over intensity variance at 256px, catalog_art.portrait_sharpness; absent when the bytes do not decode, which keeps the photo) — ONE Deezer artist portrait checked for a BLURRED face after the placeholder and blank checks. Deezer stores some acts’ portrait as a defocused art shot (Drake’s, 2026-10-15), and full-bleeding a 1000px source into an 1800x2400 card makes an obviously blurry card; every XL portrait is 1000x1000, so no size check can see it. blurry = under MIN_PORTRAIT_SHARPNESS (0.010): the photo is refused and the card drops to the next candidate (a featured act) or the next rung, otherwise silently. Calibration (219 real Deezer portraits, 2026-10-15): the one blurred face scored 0.003; the lowest sharp portrait 0.019. Raw Laplacian variance (the first idea in #2716) does not work on this source: it tracks contrast, and sharp portraits of Lady Gaga (38) and Shaboozey (4) scored under the blurred Drake (79). Panel portrait_sharpness (Media board): sharp and blurry counts plus the 10th-percentile score. No monitor: about 20 checks a day, and the expected blurry share is about one in 200, so a rate monitor would fire on noise. The failure it would catch (the floor refusing sharp faces) shows on the panel as a blurry jump and a falling p10, and costs a fall to the next rung rather than a dark surface. |
| catalog_search | utils/apple_music.py (search_catalog, songs_by_upc) | source (itunes_search; itunes_upc for songs_by_upc, the Lookup-API read by release BARCODE that resolve_music_row makes only after every search term misses – the STORM II card, 2026-09-29; the same iTunes host, so it shares the one health bucket), entity (song | album | artist, FOLDED into detail so it costs no column — query it as parse_json(detail)['entity'], which is what the panel does), ok, count (rows returned), query (the term as sent, truncated to 120) + limit (the clamped row cap) — with entity these are the TRIPLE that reaches iTunes, and therefore the measured cache KEY. limit is in it because callers pass 1, 5, 10, 50 and 200 and a cached 5-row answer cannot serve a 50-row request, so keying on the term alone counts incompatible calls as interchangeable and overstates the reachable hit rate (Codex review on #3207). query and limit keep their own columns while entity folds into detail, and the panel reads each from its own place — a live run is how that was learned. (calls - dcount(key)) / calls is the share a cache could have served, the panel’s reuse_pct; it is NOT calls-per-key, which reads 100 where the avoidable share is 99% and 1 where it is 0%. Truncation caveat: two terms sharing a 120-character prefix measure as one, which chart subjects and catalog titles never approach), duration_ms — ONE iTunes catalog search. The repo’s busiest music lookup, and until #3200 the only outbound step in the family with no event at all. resolve_music_row is built on it, so every desk card and every release link (#3188) passes through it — and song_release_date reads the same Search endpoint directly, matching on raw iTunes fields the normalizer drops, so it emits the same kind from its own call site rather than being left out of the count (Codex review on #3207); its siblings catalog_lookup and cover_art_fetch were already instrumented and it was not, so its latency, its fail rate and even its VOLUME were unknowable. That is what made “should these reads be cached” unanswerable from telemetry: the only numbers available were proxies (desk_art counts, apple_debut counts). ok is the TRANSPORT outcome and nothing else — False only when the read genuinely failed (retries exhausted, timeout, bad body), which _itunes_get reports by returning None rather than [] since #3207. A rowless answer from a working call is ok=true, count=0, reason=no_rows. Getting this split right took three passes and both wrong versions broke a shared panel: semantic-miss-as-failure told an operator to check the API key over a title iTunes does not carry, and then unconditional success would have made int_ok and media_ok read 100% healthy straight through an iTunes outage. Every success-rate consumer — ops_monitor._MISC_INTEGRATIONS, int_ok, media_ok, the last two off the shared _OUTBOUND_KINDS/_MEDIA_KINDS inventories — means one thing by ok=False, so this event means that thing too. Because that is what ok=False means, the kind IS registered in ops_monitor._MISC_INTEGRATIONS (as itunes_search), so a real outage on the repo’s busiest music lookup raises integration_unhealthy; it was left out while ok=False still meant a semantic miss, and leaving it out after the flip would have meant a total outage raised nothing there (Codex review on #3207). The rowless SHARE is the separate question and stays on the dedicated panel as empty_pct, over ANSWERED calls only (ok==true and count==0) — a raw empty COUNT is not comparable across entities or dashboard windows, since 100 of 10,000 and 100 of 100 render the same, and counting bare count==0 read an OUTAGE as “the catalog holds nothing” because a failed read carries count=0 too, so the panel also carries fail_pct in its own column (Codex review on #3207). Transport failures also still reach error triage via _itunes_get’s emit_error(source=apple_music), a 200 with no results list included. A dedicated ops-monitor finding on the ROWLESS share is the follow-on, once a baseline exists to set a floor against — the same reason #3199’s floor is deliberately loose. Watch count too: a drift toward thin answers shows before it becomes zero. Registered in the unified outbound accounting — axiom_setup._OUTBOUND_KINDS and events._VENDOR_BY_KIND (vendor apple_music) — so the overview’s outbound-calls table and the health board’s vendor rollup both count it; a kind missing from those is silently dropped from both (Codex review on #3207). An empty QUERY emits nothing — that is a caller bug, not a search. |
| desk_link | cogs/music_desk.py + cogs/music_news.py | surface (music_desk | music_news), ok (a url resolved), phase (the lane — ktt2_chatter | release), reason (no_match = the catalog held no EXACT-name artist row; link_only | no_link | no_card = which way a newsroom RELEASE went), delivered (newsroom release only: room | staged) — a desk card’s outbound LINK resolved or missed. The link half of the desk_art watch, added with the forum-chatter card (#3035). It exists because the miss is SILENT in every other signal: the link is optional, so a miss raises nothing, drops no post, and changes no *_posted field — the card simply ships without its button, which nobody sees unless they remember there used to be one. Read the ok=False SHARE per phase, never the raw count: the chatter lane posts once or twice a day BY DESIGN, so one miss is noise and a run of them says the exact-match resolver (apple_music.resolve_artist_link) has gone quiet or the lane’s subjects have moved to acts iTunes does not carry under that spelling. It is deliberately NOT music_link_missing, which is /music’s HARD degradation (a drop with no link is recomposed and counts against the music surface in the ops monitor’s DEGRADATION_KINDS); a missing chatter link costs a button, not a post. music_news’s release phase is a DECISION, not only a health signal (2026-09-12). The newsroom now posts a new release as EITHER the music link OR the card, never both (owner steer: “new releases shouldn’t have a card and music link. only a music link, probably, if it exists. it’s one or the other”), so ok=true/reason=link_only counts the drops that shipped as the take plus the Apple Music link with no card, and ok=false/reason=no_link counts the ones whose catalog row held no link, so the card shipped instead. A THIRD reason, no_card, is the double miss: no link AND no art, so the take shipped as bare text and there is no card either — it is broken out because a no_link that silently meant “nothing shipped but words” would hide catalog-and-art degradation inside the healthy-looking side of the split. Read the three together as the lane’s split. Read the SHIPPED split off delivered == 'room' (2026-09-12). The card is built before _sched_deliver branches on the stage, so a STAGING audition — a post that only ever reaches #bot-logs — runs the same route and emits the same row; the decision is also emitted before the send, so a failed send counts too. Without that filter an auditioning guild’s decisions read as shipped posts. Read them ALSO against music_news_posted carded: a release that ships its link is carded=false and is NOT a degraded post, so the carded rate on phase=release no longer means what it means on the stat lanes. A no_link share that climbs says the catalog lookup (_artwork_and_link) has gone quiet, which is silent otherwise — the post still ships, just as a card |
| music_ranking | 2026-09-29 (later the same day): the card renderer no longer draws ANY standing line, so song_attached / daily_attached now count a lookup whose result nobody sees. Read this event as lookup health only, until the follow-up removes the lookups. 2026-09-29: the ARTIST standing strip was REMOVED from every card (owner steer, after an Omar Courtz 100M card read “#437 most-streamed artist”). Only the SONG rungs still emit (song_attached / daily_attached / no_match / kworb_off / error, surfaces music_desk + music_news); a song with no song rank now ships with NO strip and emits no_match. The reasons attached / attached_youtube / feed_down / not_artist and the surfaces music_alert / market_card are retired. The text below is the history. utils/ranking_strip.py (the ONE lookup home since 2026-09-08), gated per surface by cogs/music_desk.py, cogs/music_news.py, cogs/music_alert.py | surface (music_desk | music_alert | music_news – the newsroom’s streaming-milestone card, added 2026-09-08 so a Spotify milestone keeps its rank | market_card), ok (did a standing strip attach), reason (attached = the artist strip | song_attached = a per-track milestone showed the SONG’s all-time most-streamed rank | daily_attached = a per-track milestone showed the song’s DAILY chart position | units_attached = RETIRED 2026-09-28 (a UNITS/SALES card showed its projected Billboard 200 rank; the owner removed that strip, so no lane emits it now) | no_match = no ranking found (artist off kworb’s board, or the release not in the HITS document) | not_artist = an ALBUM streams reading, whose subject is not an artist | kworb_off | error), guild_id — a card TRIED to attach a ranking STANDING strip under the figure, so a reader can gauge WHERE the number ranks. A per-TRACK streams milestone runs a LADDER (owner ask 2026-08-27): the song’s all-time most-streamed rank (song_attached), else its daily Spotify chart position (daily_attached), else the artist strip; a UNITS/SALES card carries NO strip (its projected Billboard 200 rank line was removed on 2026-09-28, owner steer); every other metric keeps the artist’s Spotify standing (their most-streamed rank + monthly listeners, per kworb). (A radio-riser card also carries a rank strip — its position is on the kworb row, so it is deterministic and rides no event.) Attaches on BOTH the Luminate streams card AND the Spotify streaming-milestone big-number card (the one home is _artist_ranking_strip). The music ALERT lane attaches the SAME strips on its Luminate cards (surface music_alert) — the artist Spotify standing on a streams card (music_alert._streams_standing) — it was missing it before, so a ‘cleared 12B streams on the year’ alert shipped a bare number card (owner report: missing ranks). The strip now also renders on the LIVE/last-call PROJECTION card, not just the settled number card: _projection_number_figure carried no standing slot, so #2636’s units rank never showed on the projection card (the common units card) on EITHER lane; it does now. Gated on the music_stream_ranking experiment, which is LIVE (default PRODUCTION) since 2026-08-25; the strip adds a PUBLIC claim that can mirror to X and can be wrong (a stale rank / a name-fold mismatch), so it stays FAIL-OPEN and a mod can still dial it OFF/STAGING on /menu (both leave the strip absent). The strip is FAIL-OPEN — a miss just drops the context and the card still ships the number — which is exactly why this event exists: an ok=False rate is the ONLY way a silent “the strip never attaches” surfaces (it raises no error, and the card looks fine). The MARKET-CARD lane attaches the SAME artist strip on a Kalshi artist COUNT market (surface market_card): a “
x_follow_change |
cogs/x_audience.py (hourly churn sweep) | ok, kind (gained|lost|sweep), reason, count; folded detail: handle, tenure_hours, followers, statuses, low_signal (per-account rows) / gained, lost, lost_low_signal (the sweep row) — the X follower-churn diff (#x-churn). x_audience records only the follower TOTAL, so it nets follows against unfollows: a day with 5 follows and 3 unfollows reads +2 and the 3 departures never appear. This walks /twitter/user/followers hourly, diffs against the x_followers snapshot table (keyed on the X user id, so a rename is not a departure plus an arrival), and names each loss. A lost row is confirmed before it is called an unfollow — reason=unfollow means the profile is still alive (real feedback about what we post), account_gone means deactivated/suspended (not feedback), unconfirmed means the per-sweep confirm cap or a failed read. The refusal path is load-bearing: ok=False with reason=incomplete|low_coverage|empty means the follower walk was truncated, and the sweep wrote NOTHING and diffed NOTHING — diffing a short list reports every follower past the cut as an unfollow, which is worse than no data. reason=seeded is the first run. HOURLY because the cadence sets the attribution precision: a departure lands between two sweeps, so an hourly window holds the 1-3 posts that could have caused it, where a daily window holds ~20. Losses also post to the master guild’s #bot-logs. Ops-monitor health, two complementary paths: the API read itself is curator_fetch with phase=followers, so it already rolls into misc_health["twitterio"] via _MISC_INTEGRATIONS with no new wiring; the SWEEP-level refusals are a separate branch feeding misc_health["x_follow_sweep"] (the per-account gained/lost rows are payload and never counted, and reason=seeded is excluded so a fresh deploy cannot flag). That second one exists because a refused sweep is the fail-open trap: it raises nothing and posts nothing, so a broken follower walk looks exactly like a quiet week while the net follower curve carries on looking healthy. Known limit — correlation, not proof: X publishes no per-viewer data to any API (twitterapi.io has no likers endpoint — /twitter/tweet/likers and /twitter/tweet/favoriters both 404, verified 2026-09-14), so this narrows a departure to the posts in its window and cannot prove the account read any of them; the reply/mention/retweet record is the only per-account evidence of a real read. Correlate to the posts in the window (tweet_posted is in the same dataset): ['tootsies'] \| where event == 'x_follow_change' and kind == 'lost' \| extend d=parse_json(detail) \| project _time, tostring(d['handle']), toreal(d['tenure_hours']), reason |
x_reply_edit |
cogs/x_audience.py (nightly sweep) | ok, post_preview (the owner-POSTED reply); folded detail: pure_cut (bool — the KPI: the posted text is a contiguous slice of the draft, i.e. the owner KEPT a beat instead of rewriting; measured 2026-08-02 over 223 pairs, pure-cut posts out-viewed rewrites ~2.7×), words_draft, words_posted, slop_score — the owner-edit-fidelity join (#ai-register): the reply cog retains each draft’s text by target tweet id (kv ns x_reply_text, 7d, cogs/x_reply_draft.py), and the nightly x_audience sweep matches the timeline page’s replies by inReplyToId (utils.x_audience.reply_edit_metrics, pure + unit-tested), emitting ONCE per posted reply (kv-stamped x_reply_edit ns, 14d). Metrics only — the draft text is never emitted. A reply with no retained draft (hand-written, pre-feature, >7d late) is silently skipped. Weekly cut-rate APL: ['tootsies'] \| where event == 'x_reply_edit' \| summarize cut_rate=avg(toint(parse_json(detail)['pure_cut'])), med_posted=percentile(toint(parse_json(detail)['words_posted']),50) by bin(_time, 7d) |
The slop_score auto-stamp (#ai-register, owner steer “every post every surface”):
emit() itself stamps slop_score (the deterministic AI-register score from
utils/slop_score.py — vendored antislop lexicon + not-X-but-Y tells, 0..1) onto ANY
event carrying a post_preview, before the fold — so every shipped post on every
surface (discourse/music/chimein/commentary/market drops/ask/…) is scored with zero
per-surface wiring, and a surface added tomorrow is covered by construction. Folded
into detail (zero field-budget cost): query parse_json(detail)['slop_score']. An
explicit caller-passed slop_score is respected (the x_reply_edit join passes it
literally so the schema scan sees the field’s in-source emit). Baseline at ship: 0.00
across every live surface. Consumed by the ops monitor’s slop_high finding (below)
and, on the eval cadence, scripts/eval_ai_register.py.
Adding a new event: call emit("your_kind", key1=..., key2=...) and add a row to
the table above + the events.py docstring. Use snake_case for kinds and fields. Don’t
include full message content (data minimization, per the constitution).
A feature isn’t done until it’s observable. Every new capability — especially a new outbound integration (an API client, a model call, an external fetch) — ships its instrumentation in the SAME change, not as a follow-up. This is mandatory and unprompted on every change going forward: apply the full bar below to anything that adds or touches a surface, integration, or outbound call — every time, without waiting to be asked. “Do we have full instrumentation for this?” should already be yes by the time the PR opens — never a question the owner has to raise. The bar, in order:
Latency + outcome, per distinct call. Wrap the outbound call with
@instrument("your_kind") (or timed_event for a block, @timed/timed_span
for a heavy flow step) so it emits one event with ok + duration_ms. Never
hand-roll a stopwatch — utils/instrument.py owns the clock. Any event carrying
duration_ms is auto-picked-up by the ops monitor’s p50/p95/p99 table; add a
ceiling to LATENCY_CEILINGS_MS (or a LATENCY_CEILING_PREFIXES rule) for each
one if a blown tail should gate, not just render. Instrument EACH distinct
external step as its own event, not one event for the whole feature — if a flow
does an API resolve, then a CDN download, then an STT call, that’s three events,
so the three failure modes stay separable in the logs (the X path’s
x_fetch + x_media_fetch + stt_transcribe split is the worked example,
#491). One bundled event hides which leg actually broke. Naming convention:
an event KIND names WHAT was called (market_fetch, claude_api); a SHARED
integration hit from many surfaces (SGO/Kalshi/Perplexity/Claude from
bet/commentator/ask/discourse/music) also needs WHO called it. Don’t rename the
kind (it’s the stable dashboard contract) — instead wrap the surface’s outermost
entry method in @scoped_surface("commentator") (or with surface_scope(...)),
and utils/instrument.py auto-stamps every nested event/span with surface= +
a hierarchical path (surface.kind[.source], e.g.
commentator.market_fetch.sgo) so the latency + error tables group by the CALL
SITE, not just the integration. Additive + fail-open (no scope → no surface
field, byte-identical to before). SPAN names (@timed) are already hierarchical
(surface.function) — keep that. Wire a new surface at its one outermost entry
method, never per call. Field budget (the Axiom schema cap): every DISTINCT
field NAME across ALL events counts against the dataset’s per-dataset field limit
(the tootsies dataset is on the Axiom Cloud plan: 1,024 fields, up from the
free plan’s 256 that #1296 hit; the whole EVENT+log stream shares this one
dataset, so it’s one budget), and AT the cap a never-seen field name SILENTLY
DROPS the whole event at ingest (a 400 naming the field, or a 200-with-failures
partial) — invisibly degrading the exact observability the ops-monitor + health
cog depend on (#1296, how curator_posted went missing on the old 256 cap). The
1,024 cap stopped the dropping, but the owner may downgrade back to free (256),
so the discipline below is kept precisely so free stays a viable fallback — after
the #1296 “full maps push” the footprint is ~146, well under 256 (a
downgrade would be clean), and the tripwire in tests/test_event_schema.py
ratchets it further DOWN as consolidation continues; don’t bump it up.
utils/event_schema.py is the single source of
truth for the field vocabulary, split into three ledgers: SHARED_FIELDS
(the reusable cross-event vocabulary — reuse a name from here when the meaning
matches: source/kind/reason/ok/count/query/phase/delivered, …,
before inventing mycog_specificthing), _EVENT_SPECIFIC (the top-level
fields a dashboard / ops-monitor / consumer actually READS — these keep a real
Axiom column), and _FOLD_FIELDS (the ~124 diagnostic singletons that no
consumer reads: emit() auto-folds any of these into the ONE shared detail
map wherever emitted — the “specific events use maps” model, so they cost NO
top-level column and can never hit the cap; query them with
parse_json(detail)['<name>'], see the _FOLD_LOG for the transition-window
coalesce bridge). To ADD a field: reuse a SHARED name if one fits; else, if a
consumer must read it top-level, add to _EVENT_SPECIFIC; else fold it (add to
_FOLD_FIELDS). tests/test_event_schema.py fails CI on an unregistered field
(the conscious reuse-or-add-or-fold gate) and ratchets the footprint DOWN toward
the 256 free fallback. Two more runtime guards back the registry so a leak can’t
slip past the static lint: emit() JSON-stringifies any dict field value
(_demap), so a
variable-key dict can NEVER flatten into per-key Axiom fields
(home_pcts.<source>/genre_tags.<genre>/context.<key> were exactly this —
put high-cardinality / variable-key data in ONE stringified dict field, not
separate fields named after the keys), and utils/axiom.py inspects the ingest
response’s failed/failures and logs (rate-limited) when Axiom rejects rows —
a silent drop becomes a visible signal. When the count nears the cap:
python -m scripts.axiom_setup --fields audits live-vs-registry (dead +
drop-risk fields), --vacuum reclaims dead-field slots (renamed/removed events),
and a wide/separable event family can route to its OWN dataset via an
AXIOM_*_DATASET env (off the main budget). The dataset is currently on the paid
Cloud plan ($25/mo, 1,024-field cap) which lifted the immediate wall; the footprint
is now under the free 256 (via the fold), so a downgrade is clean — the
consolidation levers above stay the primary tools to keep it there.
Every panel and monitor query RUNS before a push (#3218). A query Axiom rejects
renders as an EMPTY table, which looks exactly like a quiet surface, and no CI check
can parse APL. python -m scripts.axiom_setup --validate-only runs every query once
through the read-only _apl endpoint over the last hour and exits 1 on a 4xx; a
5xx or network error is reported as “not checked”, never as broken APL. The push
paths (--dashboards, --monitors, the default run) run the same walk first and
refuse to push on a rejection. First run, 2026-10-15: 338 queries, 1 rejected –
tootsies-desks / source_changes (mv-expand src=...: Axiom’s mv-expand takes no
alias; extend first). Validation does NOT catch a valid query that reads a folded
field as a column – that returns empty rows, not a 400 (see #2254 below).
Folding a field breaks every CONSUMER that reads it top-level (#2254). In
the monitor the break is silent: ev.get("kept") returns None, nothing raises,
and the branch reports 0. In APL it takes one of two forms, both measured
against the live dataset. If the name is nowhere in the dataset schema the
query FAILS – the wire panel returned HTTP 400 invalid field: "kept". If the
name IS in the schema, because another event writes it top-level or because
this one did before the fold, the query RUNS and reads null on every row:
sum(source_count) returned 0 for all four X surfaces, and by league grouped
every row under a null key. The second form is the dangerous one and the more
likely after a fold, because folding leaves the old column in the schema.
Five monitor branches and five panels shipped this way. Healthy wire_read
rows carrying kept=157 counted as an outage, and the curate routing funnel
read 0/0 on a lane routing 467 of 492 posts. So when you fold a field, GREP for
its readers first. In the monitor, read it with _ev_folded(ev, name); in APL,
read it with parse_json(detail)['<name>'] and alias it. Two guard tests hold
the line: tests/test_ops_monitor.py parses the monitor and fails on any
ev.get("<folded name>") or ev["<folded name>"], and
tests/test_axiom_setup.py scans every shipped panel and monitor query for a
folded name used as a column.
emit_error(source=..., exc=..., recoverable=...)
so error-triage groups them. Set recoverable honestly (user-impacting →
False). At framework boundaries, emit the application exception rather than a
generic transport wrapper. The global slash-command handler unwraps discord.py’s
CommandInvokeError, so Axiom groups the real cause and the bot-log classifiers can
recognize database and prompt failures.Silent / fail-open degradation is the trap. If the feature is fail-open (a
miss → degraded-but-not-crashed, like embeddings → keyword fallback, X
enrichment dying, a market source going dark), an ok=False raises NO
exception, so error-triage never sees it. Wire it into the ops monitor’s
integration-health rate tracking (an OkRate keyed by its dimension +
an integration_unhealthy-style finding) so the silent degradation surfaces.
This is the single most-missed step.
“Keyed by its dimension” is the whole load-bearing part (#3040). One event
kind that carries many independent reads must NOT share one rate, or a healthy
sibling hides a dead feed. Measured on #3042: kworb cut its US YouTube page from
100 rows to 20, the yt_us_weekly read failed on every attempt for a full day,
and the monitor said nothing — because chart_fetch was one bucket for every
kworb page. Over that window the pooled rate read 4.8% failing (28/578)
against a 60% gate, while the dead feed alone read 92% (24/26). The
telemetry was already perfect; only the bucketing was wrong. _PER_SOURCE_MISC
and _PER_QUERY_MISC in scripts/ops_monitor.py are where a kind declares
which dimension splits it, and health_query_key collapses opaque per-entity
ids (track_<id> → track) so the map stays bounded — 227 distinct
chart_fetch query values over 7 days become 17 real feeds. chart_lookup is
keyed per source since #4175 (genius / musicbrainz / omdb / tmdb), so a
dead tmdb weekend-budget read (#4120) can raise integration_unhealthy.
events.py
docstring, and (if it changes what the QA routine watches) update the ops
monitor section. The scripts/ops_monitor.py deterministic core is pure and
unit-tested — add a test for any new finding/health branch.The litmus test: if this surface started silently failing in production at 3am, would a graph or a flagged finding show it by the next ops-monitor run? If not, it isn’t instrumented yet.
Field coverage is part of the feature too. When you add or change an outbound
integration, hand the model every answer-valuable field the upstream returns (the
data-minimization rule governs telemetry + her replies, NOT the in-context input),
and make each surfaced field self-describing in its rendered block or tool
description (a legend/label, so the model knows what the value IS). docs/INTEGRATIONS.md
is the per-API coverage ledger: it lists, for every integration, the fields we
surface and the fields we deliberately omit (with the reason). The rule is 100%
of the answer-valuable fields, or an explicit entry there saying what’s missing
and why — keep it in sync when you touch an integration.
Railway dashboard queries: filter logs for the EVENT prefix, then parse the JSON
suffix. Typical queries: count of event=command per minute, p95 of duration_ms
where event=claude_api, sum of output_tokens where purpose=ask for cost tracking.
Axiom time series render: counts are BARS, everything else is a LINE (owner
steer). scripts/axiom_setup.py sets the variant per panel automatically: a
TimeSeries panel renders as bars when EVERY summarize output column is a bare
count() or countif(...); a p99 (or any sum, percentile, gauge, or computed
rate) keeps the default line. The variant rides on
query.queryOptions.aggChartOpts – a JSON map keyed by the column descriptor
({"alias":"count_","op":"count"} for a bare count) valued {"variant":"bars"}.
_bar_variant_opts derives it from the APL, so a NEW count panel gets bars with
no extra wiring; tests/test_axiom_setup.py pins the count/line split by panel id.
scripts/ops_monitor.py + .github/workflows/ops-monitor.yml are the bot’s one
automated QA routine, running twice daily (cron: 0 8,20 * * *, + manual dispatch).
A deterministic pass pulls the last ~12h of Railway EVENT logs (GraphQL, via
RAILWAY_API_TOKEN/RAILWAY_SERVICE_ID) and flags both halves of QA:
THE MONITOR READS ITS WINDOW IN SLICES, AND SAYS SO WHEN IT CANNOT (#3603). Until
2026-09-22 fetch_events_axiom ran ONE query carrying sort by _time asc | limit 65000.
Two faults, both silent. The APL limit never bound: an Axiom matches response is capped
at 50,000 rows whatever the query asks for (measured – limit 65000 and
limit 150000 both return exactly 50,000). And the ASCENDING sort meant the 50,000 kept
were the OLDEST, so the most recent hours of every window were discarded – measured
2026-09-22, a 12h window ending 06:19Z was read only to 00:42Z, losing 5h37m. The monitor
exists to catch a surface that just went dark, which is precisely what sat in the part it
threw away. The run still reported success.
The window is now read as AXIOM_WINDOW_SLICES (8) consecutive time slices, each its own
query with its own 50,000-row budget, over ABSOLUTE boundaries resolved from one clock
read (a now-Nm string is re-resolved per request, so each slice’s start drifted past the
previous slice’s end by its fetch time and dropped ~25-30s of events into the gaps), each sorted DESCENDING (so a slice that does
overflow drops its oldest rows, not its newest) and re-sorted to ascending at the end for
LatencyStats.trend(). Measured effect on one live window: 50,028 events read before,
117,897 after, and 60 findings where the truncated read produced 53 – seven real
findings had been invisible. A slice that still fills the cap inserts a synthetic
_window_truncated row that aggregate reads into window_cut_at; evaluate turns it
into a HIGH window_truncated finding and render puts a PARTIAL WINDOW banner above
everything, because a truncated report cannot say a surface is quiet, only that it did not
see it. Volume grew from 94,273 to 116,634 events per 12h window in a single day, so
AXIOM_WINDOW_SLICES is the knob to raise; the durable fix is to push the per-source
rollups into APL summarize so the monitor stops pulling raw rows.
Three robustness rules came out of testing this, each from an observed failure. A slice
body runs to ~150MB and http.client.IncompleteRead killed the read twice, so
_axiom_slice RETRIES – without it the whole run falls through to the Railway tail.
That fallback is now MARKED with the same truncation marker, because the tail is ~800
events against ~120,000 and an unmarked report renders identically to a healthy one with
every surface looking quiet. And AXIOM_TOTAL_ROW_CEILING caps the rows held across all
slices: slicing removed the implicit ceiling the 50,000-row response cap used to give, and
~14MB per 2,000 parsed rows puts an unbounded window near a gigabyte resident, where the
job is killed and reports nothing at all.
One cut slice no longer drops the whole run (#4359). Measured 2026-10-17 from a
session: IncompleteRead cut 6 of 32 slice reads at 8 slices and 2 of 16 at 16 slices.
The cut came after 47-88 MB and 3-5 s, while bodies up to 156 MB read clean, so more
slices alone do not remove it. With one retry, about 1 run in 4 could lose a slice twice
and fall back to Railway. Now _axiom_slice:
AXIOM_SLICE_ATTEMPTS (4) tries through utils.retry, with a
2s/4s/8s backoff;AXIOM_SLICE_SPLITS = 2), each piece with its own tries;A gap goes on the truncation marker as gaps (a list of start..end ranges).
aggregate reads it into window_gaps. evaluate adds a MEDIUM window_truncated
finding that names each range, render adds a PARTIAL WINDOW banner, and the
chart_capture_stale “no row” checks stay silent, as they do for a cut window. Only a
run where EVERY piece fails raises, so a real Axiom outage still falls back to Railway
and names the error.
The fallback ran on EVERY scheduled run from at least 2026-09-22 to 2026-10-16 (#4159).
The workflow passes AXIOM_DATASET: $, that secret does not
exist, so the variable was SET to “” and os.environ.get(name, "tootsies") returned “”.
The query became [''], which Axiom refuses with 400 “can not build key, key missing or
empty”. The reports read 0 to 8,268 Railway events in place of about 120,000. Two fixes:
AXIOM_DATASET (ops_monitor, competitor_scoreboard, axiom_setup,
config) now calls utils.axiom_dataset.axiom_dataset(). It reads an empty value as
unset and returns the one constant DEFAULT_DATASET. The module is stdlib only, so
it imports before any package install. Use or for any env value a workflow passes
through from a secret. The workflow no longer passes AXIOM_DATASET at all.axiom_error, with the HTTP body via
axiom_http_error) and the real Railway row count. The window_truncated finding and
the banner then name the failed read. Before this they said “read only 0 events” and
“a slice still overflowed, raise AXIOM_WINDOW_SLICES”, which is wrong advice for a
failed read.The CI posting-gap query got 403 “forbidden” in the same runs. The session’s key, which is
the same as the Railway service’s AXIOM_API_KEY, gets 400 for [''] and 200 for
['tootsies']; an invalid key gets 403 before the dataset is checked. So the
AXIOM_API_KEY repository secret is probably not a key that can query the dataset.
Confirmed 2026-10-17. With the dataset fix on main, a manual run still got 403 on every
query (run 38996637814, 3,894 Railway events). The secret held a different xaat- key.
The owner pasted the Railway AXIOM_API_KEY value over the secret, and the next run read
Axiom with no 403 (run 38996944596, 142,927 events). A malformed key (quoted, cut short,
“Bearer” pasted in) gets 401 “token not supported”, not 403. So a 403 here means a
well-formed key that Axiom does not accept. Overwrite the secret from Railway first.
LATENCY_CEILINGS_MS + LATENCY_CEILING_PREFIXES; the gate is p99, not the mean, since a healthy average can hide a blown tail). Latency is self-contained in one LatencyStats class and tracked uniformly for every instrumented surface — LLM purposes AND external HTTP clients (Perplexity/markets/TTS/STT/video/image/gifs/catalog/link-enrich/link-check/GitHub/Railway/slash commands, anything carrying a duration_ms); claude_api is no longer special-cased, it flows through the same map keyed by purpose (its token stats stay separate in PurposeStats, which just composes a LatencyStats). The report renders one unified p50/p95/p99 table with a trend column, and flags two things: a latency finding when a gated surface’s p99 crosses its ceiling (NOT the DB: db_query is a slow-call sample with no denominator, so the DB gets its own count-based db_slow finding and table instead, #2257), and a latency_trend finding when a surface’s tail is climbing within the window (earlier-half vs later-half p99, an early-warning before it crosses). Plus hallucinated-link rate per surface, ungrounded chime-ins (0-search posts), failed slash commands, rate-limit pressure, and video transcription health — the video_transcribe resolution fail-rate (no_info/transcribe_failed over real attempts, excluding provisioning/too_long skips); a high rate flags video_unreachable, the datacenter-IP-gating signal (the #1 risk the video feature shipped with), and a video-health line renders every run so the hit-rate is watchable even when it’s fine. Plus integration health — the live success rate (cache hits / no-op guards excluded) of link_enrich per platform, market_fetch per source, the Perplexity hedge rate per surface, embedding per model, openai_api per purpose (the GPT TEXT backend, #openapi-parity), and the ScrapeCreators /ask tools (comment_read/reddit_search/social_lookup/social_search, #1272) rolled up under one scrapecreators key (they share the client) — with benign routing misses excluded (unprovisioned/not_found/empty_query/… don’t count, only a real fetch_failed/bad_shape/HTTP error does, so a user typo can’t false-flag the API): a high fail/hedge rate flags integration_unhealthy / pplx_hedged (a silently-failing external surface — X enrichment dying to guest-token gating, a market source going dark, Perplexity punting, OpenAI embeddings down so semantic recall silently drops to keyword/FTS, a GPT-routed surface silently failing over to Claude — these ok=False/hedge outcomes raise no exception so error-triage never sees them), with an Integration-health line rendered every run. The GPT→Claude provider_fallback count is rendered every run (per purpose), render-only (a graceful fallback isn’t itself an incident — the openai_api fail rate is the flagged signal), so the failover rate is watchable the moment a surface is opted onto GPT. Plus the The Odds API metered budget (#725/#725) — the key (a monthly credit allowance, plan-dependent so verify live via /debug/usage rather than trusting a number here; observed ~100k/month as of 2026-06, was 20k earlier) is the SGO-down backstop for /bet odds + live scores, and it has no usage endpoint, so the ops monitor tracks the lowest credits_remaining off its market_fetch events: a the_odds_api budget line renders every run, and a quota_low finding (high) fires under ODDS_API_CREDITS_LOW (2,000) so a runaway burn (a sustained SGO outage driving the live-scores fallback) is caught before the key exhausts mid-month. #3583 added the BURN RATE on top of the balance, because the balance reads healthy right up to the day it empties: the 20,000 credits emptied on 2026-09-20 and nothing reported it until the 401s stopped the NFL Sunday slate being priced (#3552), even though the balance had been falling for a week. The monitor now keeps the window’s credits_used min and max (a FOLDED field, read with _ev_folded – a top-level read returns None on every real row and the finding never fires), turns the delta into credits/hour over EVENT_WINDOW_HOURS, and projects the month end with project_odds_api_spend against the hours left in the calendar month (hours_left_in_month; the plan resets on the month, not a rolling 30 days). Three findings: the_odds_api:burn_rate (quota_burn, high) when the projection passes ODDS_API_PLAN_CREDITS (20,000 – verified against 30 days of live telemetry, where max(credits_remaining) is 19,999 and max(credits_used) reached 20,007); the_odds_api:reserve (quota_low, medium) when the balance falls under ODDS_API_RESERVE_CREDITS (5,000, the enhancement reserve utils/the_odds_api.py holds back for the /bet backstop) but is still above the 2,000 floor, so the two never double-report; and the_odds_api:quota_dead (integration_down, high) counting error=quota_dead refusals, since the client sleeps 6h after a 401 and that silence otherwise reads as health. A projection needs at least ODDS_API_BURN_MIN_SPEND (50) credits in the window – #3578 took the steady state to near zero on purpose, so a trickle must not project a month. Replayed against the real September burn it first fires on 2026-09-13 and stays quiet on 09-11 and 09-12, which is seven days of warning ahead of the exhaustion. A the_odds_api burn line and a the_odds_api quota_dead line render every run; scripts/axiom_setup.py carries a credits-remaining time series (the SLOPE is the signal), a per-surface burn table, and a monitor on the 5,000 reserve line.discourse_scored/music_scored); a surface whose recent posts mostly land
below SCORE_FLOOR (the bot’s own 0.6 ship gate) is flagged low_quality, and
must_post posts that shipped under the floor are called out (users saw them). Silent
degradation (canned fallbacks / skipped slots / missing links) is flagged from the
*_fallback / discourse_skipped / music_link_missing events.(surface, model)
for any surface that ran on >1 model in the window (the A/B signal — did the GPT
swap score worse than Opus on that surface?), off the model field the scored events
carry; a “LLM spend by provider” section rolls the claude_api/openai_api token
totals up to an anthropic vs openai split with a rough $ estimate (raw tokens are
the exact signal, _MODEL_PRICING the approximate/env-updatable rates) so “is the GPT
opt-in moving spend off the Anthropic bill” is answerable every run; and a low-volume
all-fail floor flags integration_unhealthy when a GPT surface fails 100% below
the 5-attempt rate floor (LOWVOL_FAIL_MIN) — the class the GPT-vision-400 (2 calls/day)
slipped through invisibly.slop_high, #ai-register) — per posting surface (event kind),
the mean of the slop_score auto-stamp (see the event-stream section) over the
window; ≥ SLOP_MEAN_CEILING (0.15) with ≥ SLOP_MIN_POSTS (5) posts flags medium.
Baseline is a flat 0.00 across all live surfaces, so a sustained mean here means the
register regressed (a prompt change, a model flip) — the runtime, twice-daily twin of
the scheduled eval_ai_register pass. Kind-agnostic by construction (any event with
a post_preview participates), so new surfaces are covered without ops-monitor edits.source_blocked) — per reason (blocked = a 403/451 refusal, shape_change = a 200 that parsed to near-zero rows, partial_credits = a 200 whose rows had no artist credit, i.e. a page read mid-publish on flip day, undated = a 200 for a DATED page that prints no week, i.e. the template lost its date line; the chart-history walk’s quiet reads never record a block), with the longest CONSECUTIVE run any one chart hit in the window and the charts still failing at the window’s end (per-chart, keyed by source so a stand-in can never mask the primary). blocked is always high: a refusal is a decision. shape_change is high when a chart ran ≥ BILLBOARD_SUSTAINED_RUN (3) consecutive failures or is still dark at the end — it lost data for real, whatever the cause — and medium when every affected chart recovered sooner. That split exists because the old finding called every shape_change a redesign and sent the operator to fix parse_chart; measured on the 2026-09-01 weekly refresh, a handful of genre charts served an empty 200 for ~40 minutes while hot100 stayed fine, then all recovered, and the unchanged parser reads today’s page 200/200. That is Billboard republishing pages, not a markup change, so the high wording now says “only some charts / the parser is fine” unless EVERY chart is dark and none recovers (#2824). partial_credits follows the same shape with its own wording: medium when every affected chart completed within the window (Billboard finishing the page, no action), high only when a chart is still partial at the window’s end — a row Billboard prints with no credit would keep that chart dark all week, and the finding says the guard then needs a threshold (2026-09-09).deliver_failed, #2063) — per surface, the count of
post_deliver_failed events (a gate-passed, non-deduped unit whose _sched_deliver
returned False). Expected zero, so ANY count flags high: the surface is composing
posts (paying the model calls) and the room never sees them, and since a
nothing-delivered slot is not recorded, the slot re-fires and a broken room config
repeats every tick. Complements surface_dark: that finding reads judge scores, so a
surface that composes and gates fine but cannot SEND still looks healthy there — this
is the branch that catches it.wire_empty, #3479) — per platform of curator_fetch phase=user, high
when at least WIRE_EMPTY_MIN_FETCHES (5) fetches landed in the window and
WIRE_EMPTY_RATE (80%) or more of them returned zero posts with ok=True. One quiet handle
returns 0 legitimately, so this reads the platform, never a handle. Measured 2026-09-18
19:40-19:59Z: the X API answered {"status":"success","tweets":[]} for every handle (10 of
12 fetches), every wire desk read as a quiet day, and the integration ok-rate stayed 100%.surface_dark) — the louder sibling of low_quality: when a
scored surface lands ~everything below the ship floor (DARK_SCORE_RATE 90%+ of
DARK_SCORE_MIN+ posts) it isn’t “scoring low”, it’s shipping nothing — users see
silence. Flagged high and cause-agnostically across every scored surface
(commentary/discourse/music/…), distinct from low_quality (medium) so it doesn’t
read as “her posts are mediocre” — the misattribution that let the #697 self-gate-drop
hide (the scorer truncated every commentary score to 0.0, so 0 posts shipped). Takes
precedence over low_quality for the same surface (no double-report). The companion
Self-gate health line renders every run (ship-rate per surface + each *_score
judge’s truncation rate) so the blackout signal is watchable even at a healthy 100%/0%.
A false blackout is the mirror risk (#2254): five branches here read a field that
emit() folds into detail, so they saw None on every production row. Healthy
wire_read rows carrying kept=157 counted as an outage, and the curate routing
funnel printed 0/0 on a lane routing 467 of 492 posts. Every folded read now goes
through _ev_folded(ev, name), and a guard test parses the monitor and fails on a
direct ev.get/ev[...] of any _FOLD_FIELDS name. Same failure in the
dashboards: five panels grouped or summed a folded field as a column, so they
read null on every row or failed outright; tests/test_axiom_setup.py guards
those. Every other query in the repo is guarded too (#2067):
tests/test_apl_folded_reads.py cuts each APL query out of the Python files (the
ops monitor, the live-log pass, the competitor scoreboard, the dry runs, and the
docstrings) and out of CLAUDE.md, docs/, COVERAGE.md and the skills, and fails
on a folded name read as a column. The silent case is the common one: on
2026-10-24, 102 of the 251 folded names were still columns in the dataset schema
(left there from before their fold), so a bare read of one runs and returns no rows.
The first scan found two copied doc queries doing it (handle in
docs/BILLBOARD_SOURCES.md, followers in docs/X_GROWTH_ROADMAP.md). The
detector (tests/_apl_lint.py) is shared by both guards. An ad hoc query that a
session types is still not guarded: read the utils/events.py index first.surface_silent high, surface_post_drop medium,
#2936) — surface_dark and lane_dark read judge scores. A surface that stops
ATTEMPTING has no scores and no events, so it is absent from every map and looks like
a surface with nothing to say. The commentator posted nothing from 2026-10-04 19:30Z to
2026-10-15 and nothing flagged it. This check does not read the 12h window. It runs one
extra Axiom summarize (fetch_posting_gap_findings): the count of every *_posted
event per hour over POSTING_LOOKBACK_DAYS (28). posting_key splits each event on
surface and on guild_id: one guild’s healthy surface must not hide the same surface
dead in another guild. The artist curator emits curator_posted with source=roster
and no surface stamp, so the query derives surface=artist_curator from that
source. The pure core is
posting_gap_findings(hourly, now). It is table-driven, so a new surface is covered
with no wiring.
surface_silent: the surface has SILENT_MIN_POSTS (10) posts on
SILENT_MIN_ACTIVE_DAYS (2) or more days, and its last post is older than BOTH
SILENT_MIN_HOURS (36) and SILENT_GAP_FACTOR (1.5) times its own longest gap
between posts in the lookback. The longest-gap rule keeps bursty surfaces quiet (the
bet board and the awards desk go days between posts). It does not need daily posts:
the commentator posted on only 2 days before it stopped.SILENT_FLAG_HOURS (72h, 6 runs) after the silence crosses its
limit, then stops. The issue filer dedups against OPEN issues only, so a finding
that kept firing would re-file a retired, off-season or switched-off surface every
12h once someone closed its issue. In the replay every real outage flagged on its
first run, so the 6 runs are margin for failed monitor runs. The cost: a silence
older than its window is not reported again. On the day this shipped, the three
live outages were already past their windows, so they are tracked in #2936 and
#4070 instead.surface_post_drop: the surface still posts, but the last 24h is under
POST_DROP_RATE (15%) of its median day over the POST_DROP_BASELINE_DAYS (7)
before, with that median at POST_DROP_MIN_DAILY (20) or more. It does not model
a weekly rhythm. On 2026-09-17..10-14 the quietest full day of any surface over the
volume floor was 56% of its median (music_desk, Wed 09-30), so the 15% bar is about
4x below a normal quiet day. Same-weekday history would give only 4 samples in the
lookback.music_news_posted silent 09-24 to 09-29 (it scored a few drafts a day and shipped
none for a week).commentary_posted silent from 10-09.
Before the guild split, the first curator outage was invisible: the other guild’s
posts kept the combined key current.source_dark high, #2948) —
source_thin is a non-score phase, so the quality and decline findings drop it, and
the source fetch reads ok=True. surface_silent needs a post history. So a surface that
runs, finds too little on every run, and has never posted had no finding (the releases
board, #2947). The fetch_posting_gap_findings query now also returns every
phase == source_thin row, so this costs no extra read. source_thin_posting_key keys
each thin row on the surface’s *_posted twin (music_releases_scored ->
music_releases_posted) per guild, with no surface split. The pure core is
source_dark_findings(thin_daily, posted, now).
SOURCE_DARK_MIN_DAYS (8) or more days, and had a thin run in the last 2 days. On
2026-09-18..10-18 the healthy releases board (the only source_thin emitter) had a
longest gap of 4.4 days between posts; it posts on a weekly release cycle, so 8 days
is more than one release week.SOURCE_DARK_FLAG_DAYS (3) more days, then stops, for the same reason
as SILENT_FLAG_HOURS. The cost: a surface that was already thin for more than 11
days when it first runs is not reported. A surface that posted in the lookback and
then went thin is surface_silent’s case.fetch_coverage_findings runs three
extra Axiom queries, each with its own lookback. Each check fails open on its own.
card_art_weak (medium; high at 50%): per market_subject surface over 7 days, the
share of card builds whose art is found (an unverified web photo) or none (no
art), cache hits included. It fires at CARD_ART_WEAK_RATE (20%) over
CARD_ART_MIN_CARDS (10) cards. The card_art integration rate counts found as a
success and flags none only at a blackout (60%), so it read healthy. Measured
2026-10-09..10-23: alert, drop and market_outcome had 0 found and 0 none in 52 cards
before 10-18; in the 7 days to 10-23, market_outcome 64% (32 of 50), alert 25% (29 of
114), drop 21% (13 of 62); music_alert 0% of 457 and bet_board 2% of 61. Pure core:
card_art_findings(counts). Panel: card_art (now split by surface).story_age_stale (high): a shipped music_news_posted (delivered room or x_only)
whose event is older than the news-age limit: STORY_AGE_EVENT_DAYS (10) for chart,
first_week, cert and milestone (the gate’s _EVENT_AGE_KINDS, a test pins the copy),
STORY_AGE_DEFAULT_DAYS (3) for the other kinds. The event date comes from
detail.age_read, the gate’s own Grok read (ledger row above). stated_event_date
reads it in four tiers, each taking its latest date: after an EVENT word (occurred,
happened), so a recirculated story (“occurred on June 30 … reported again on
October 21”) keeps its event date; after a FIRST report (“first reported on”,
“earliest posts appeared”, the verb next to the date and no other date between); after
any “when” word (appeared, reported, released, since, …); any date. “released”,
“certified”, “achieved” and “since” stay out of the top tiers because they often date a
background fact (“certified platinum in 2019 … announced on October 20”). A date after the post
is skipped (a tour leg, a release day); a date with no year takes the year nearest
the post; a month alone reads as its last day; a bare year counts only right after
“occurred” or “happened” and before the post’s year, and reads as December 31
(“his first No. 1 since 2019” is background). Each guess makes the story as young
as the words allow. Lookback 7 days.story_age_undated (medium): since the first post that carries age_read, under 50%
of 10 or more shipped posts carry a read with a date. The gate fails open, so this is
the sign that it stopped reading.is_stale_recap(record=...), live Grok) on
145 of the 274 distinct stories shipped in the 7 days to 10-23. 143 reads had a date.
14 flagged (on the final parser), including the #4671 posts (Janice STFU, Sunflower, Future’s 53 plaques,
Amy Winehouse’s Frank), Snoop Dogg’s 1994 No. 1 and a 2025 Kentrell stream milestone.
Three were release stories 3 to 4 days old (borderline). The first parser took the
first date and flagged 4 more on future dates; the second took the first ANCHORED
date and picked background dates (coordinator review on #4696). The prose read stays
ambiguous in principle (each rule has a counter-example); the structural fix is a
dated line in the gate’s own prompt (#4704).orders_gated (medium; high past 7 days): per guild over 28 days, auto_order gated
for more than ORDERS_GATED_DAYS (2) with no filed order since, and still gated in the
last 2 days. auto_order is emitted per attempt, so the run is not cut at a gap in
attempts (the #3523 run had none from 10-17 to 10-22 while the in-flight limit stayed
full); the detail states on how many distinct days the gated attempts were seen. The
daily cap counts filed orders and resets each day, so the cause is the
kitchen switch, a failed deploy or the in-flight order limit. Measured: 846 gated per
guild in 2 guilds over 30 days and 0 filed since 2026-09-20 (#3523). Since #4684 the
order sweep burns a PREPPING order with no PR after 7 days (order_state via=expired),
which clears the stuck orders; this finding stays for the next cause. Panel: auto_order_reason (now with
last).card_art_weak for market_outcome (high), alert and
drop (medium); orders_gated high for both guilds; no story_age_* finding, because
no row carries age_read before this change deploys.repro (APL): ... under each finding. The query is
repro_apl(finding), a pure function. The common kinds get an exact filter: error
(and forwarded log: errors), deliver_failed, command_failure, latency and
latency_trend (per latency_label prefix), scorer_truncated, reasoning_exhausted,
slop_high, market_movers_dropped (market_filtered), quota_unreadable,
surface_silent and surface_post_drop (the posting_key over the 28-day lookback, by
day), and the three coverage findings above (each over its own lookback). Every other
kind gets an APL search for the parts of its command, split on : and /. The
window and deploy findings get no query. The report also tells the judge to open at
most FILE_LIMIT_PER_RUN (2) NEW auto-eval issues per run, highest severity first,
to comment on an open duplicate instead, and to copy the repro line into each issue.
The workflow prompt (ops-monitor.yml) is unchanged; the judge reads these rules in
the report, like the perf rule in the optimization section.
File first list (#4799). file_first(findings)
is pure. It lists the HIGH findings grouped by SOURCE (the command up to its first
: or /; the doubled integration-health key billboard_billboard folds to
billboard), at most FILE_LIMIT_PER_RUN groups, in report order. Why: on
2026-10-26 08:06Z every Billboard read had failed for 14 hours and the report held
three HIGH Billboard findings, but the judge spent both slots on a MEDIUM riaa
finding and a perf issue, so the outage got no issue for 17 hours. The same run’s
source_blocked text also said a dark-everywhere Billboard means parse_chart
needs updating; it now says to check a page for rows first, because Billboard served
its EMPTY chart page (normal template, no rows, no week line) to every client.auto-eval issues were open, none had closed since
2026-10-17, and 3 of 63 carried an Axiom query. A session that picks up an issue had
to find the query in this file first.radio_title_blind (#4180): the radio_standing no_title share is 85% or more over
at least 10 lookups (RADIO_NO_TITLE_SHARE, RADIO_MIN_LOOKUPS). Baseline 50% (33 of
66, 2026-10-20). The cause is a Mediabase title form the stripper does not read.stream_lookup_hollow (#2687): per surface, the stream_lookup ok rate is 10% or less
over at least 5 lookups and 3 distinct releases (STREAM_LOOKUP_*). The finding states
the songstats_fetch rate beside it, because a green API with a hollow lookup means an
ingestion lag or a wrong-track resolve. The subject floor keeps one release looked up
five times from firing alone. Dry run on live data, 2026-10-25: it fires for
music_desk (0 of 15 over 5 releases in the 12h to 2026-10-24 00:00 UTC).printed_units_dark (#3631): at least 3 printed_units_sweep rows (a due settled units
reading) and zero restated (PRINTED_UNITS_MIN_SWEEPS). The finding names the
billboard_story rate so the reader knows whether the fetch or the matcher broke.
Known false case: a settled release outside the printed top 10 has no printed figure.
Not done as an Axiom monitor: these lanes post weekly or a few times a day, so a 12h
ops-monitor window with a share or a zero is the alarm. A fixed Axiom threshold would
fire on every quiet week.integration_unhealthy, #3229) — every rate-based
integration_unhealthy site (link_enrich, market_fetch, the misc integrations,
embedding, the OpenAI text backend, x_media_fetch) takes its severity from
_integration_severity. It is high at a fail rate of INTEGRATION_DEAD_RATE (0.95)
or more over INTEGRATION_DEAD_MIN_ATTEMPTS (20) or more calls, else medium. SGO
failed 100% for weeks and read medium, the same as a source at 60%. A dead primary
means the fallback carries every call. The low-volume all-fail branch stays medium.typesafe health line, #4310) — every
typesafe_api event feeds misc_health["typesafe"], so a failing key or an outage
raises the normal integration_unhealthy finding at the 60% floor. Breaker
short-circuits count as fails, because the caller sees the same None. Jev spend lands
on its own typesafe line in “LLM spend by provider” (_provider_of), at
_MODEL_PRICING["jev"] = $0.042/1M input, output free. typesafe_shadow feeds
typesafe_shadow:<purpose> agreement lines, which are REPORT-ONLY: no floor and no
finding (owner direction). Jev may decide live while its old grader is the bug being
fixed, so disagreement is expected. Jev FAILURES are gated by the typesafe health
line and the Axiom monitor, not by agreement.lane_dark, #2056) — surface_dark keys on the SURFACE, and a
surface can be several lanes wearing one label: the music desk runs ten
(chart / called_shot / reconciliation / reveal / milestone / standings / chart_run /
watch_chart / radio / projection) behind a single music_desk. One lane can stop
shipping entirely while the surface stays green because its siblings still post — the
“would a graph show it by 3am” test failing for anything below surface granularity.
So a surface listed in LANE_PHASE_SURFACES is ALSO aggregated per phase
(agg.lane_quality, keyed <surface>/<phase>) and flagged medium — the room is
still getting the other lanes, so it’s a degraded surface, not a blackout. Same
rate/minimum as surface_dark, so a sparse lane (reconciliation fires only when a
called shot comes due) simply never reaches the sample floor and stays quiet: it
under-fires rather than crying wolf. Suppressed when the whole surface is already dark,
so one outage isn’t reported ten times. Four surfaces stamp a lane phase and are
covered: music_desk (chart / standings / reveal / tracking / milestone / radio /
…), cinema_desk (box_office / streaming / release / scores / awards / tv_news),
music_news (chart / first_week / touring / cert / release / market /
milestone) and, since 2026-10-24, music_alert (#2167: 35 lane phases in 30 days,
itunes_jump / itunes_debut / line_move / settle / the crown lanes / …). Its lanes are
bursty, and a quiet lane has no scored rows, so it cannot read dark: replayed on 30
days in 12h windows, 0 windows flagged. Its settle markers are NON_SCORE_REASONS. sports_desk, pop_desk and cinema_news stamp none, so they cannot
join until their scored emit carries one. Membership is verified against REAL events,
never a grep — #2059 shipped claiming only the music desk qualified, which was wrong
because the check grepped for phase= on the event name’s own line and both of the
others emit across several lines (#2065).market_movers_dropped, #3884) — the finding
watches the silent candidate exclusion (#world-cup-spike), but it used to skip a
hard-coded list of RENDERED reasons, so every new correctness guard flagged until
someone listed it. On 14 days to 2026-10-24, 14 reasons outside that list would have
filed a false finding (soft_price_tier in 28 of 29 windows). It now flags only
MARKET_FILTER_FLAGGED (horizon medium, unwatched_sport low), unconfirmed_leg
(own branch) and MARKET_FILTER_CHECK_FAILED (act_check_failed,
leader_check_unavailable: a model check gave no verdict, medium). Every other reason
renders in the “Filtered markets” line only, including a reason added later. Dry run
on the live 12h window of 2026-10-24: 10 findings before, 2 after (horizon,
unconfirmed_leg).genre_tag_blind, medium, #3165) — genre_debut_tag is
ok=True on every outcome but breaker_open, so its itunes_genre_tag rate reads
healthy while the tag drops every candidate. genre_tag_findings(agg) reads reason:
genre_debut_tag:deezer: GENRE_DEEZER_DARK_MIN (3) or more DISTINCT albums where
Apple named a kept genre (only those read Deezer) read genre_unconfirmed, and no
album read tagged or genre_disagree. Distinct albums, because
genre_unconfirmed retries every slot while a confirmed album is cached and emits
nothing, so one unindexed album would otherwise flag (Codex review on #4748).genre_debut_tag:no_match: an album reads no_match GENRE_NOMATCH_MIN (12) or
more times in the window. A search with rows holds the album 6 hours (about 2
lookups a window), so 12 means the search fails on every slot. On 30 days to
2026-10-24: 7,075 no_match rows from 17 albums, median 41 per album per window,
among them R&B and hip-hop debuts (Victoria Monet, Tinashe, LUCKI). The surface bug
is #4734.
“Candidates but zero tagged” was not built: most debuts are pop, so 33 of 61 windows
had zero tagged with nothing wrong.lane_stale_upstream, high, #3357) — the gap
that let the Pollstar touring lane and the Mediabase radio lane post nothing from
2026-09-01 to 09-15 with no finding, no error and no unhealthy integration. Every
other signal read green: pollstar_fetch ran 520 times in three days at a 100% ok
rate, because the feed DID answer — it answered with the week of 8/14. The freshness
fence refused each board correctly, but a refusal that only writes a score=0.0
marker is not a finding, and surface_dark could not see it either: music_desk ships
hundreds of posts a week from its other lanes, so the roll-up stayed healthy while two
lanes were dead. Counts stale_chart markers per surface/phase and fires at
STALE_LANE_MIN (8) in a window — the desk reads each chart a couple of dozen times a
day, so a real stall clears the bar in one window and a single transient yield does
not. Now that the reader follows Pollstar’s weekly id rotation itself, a sustained hit
means the upstream genuinely has not printed. The lesson is the general one: when a
fence goes quiet to protect correctness, something has to say it is quiet, or the
quiet becomes the outage.desk_link rows for surface=music_news, phase=release,
delivered=room (a STAGING audition is not counted), counts no_link AND no_card
as link misses, and renders on every run once a drop has shipped. The
release_link_dark finding that used to sit on these counts was DELETED in #3215:
measured 2026-10-17, the lane shipped 16 room drops in 30 days, all with a link, and
at most 4 in one 12h window, so its 5-drop floor never fired. A floor of 2 would be
reached in about 3 of 60 windows, so an outage would take about 10 days to flag. At
about one drop a day, a person who reads the block sees a blackout as fast.gate_dark, #2214) — the sharper twin
of lane_dark, for the failure lane_dark structurally cannot reach. Every gate in
the debut family (apple_debut, spotify_debut, watch_new_entry, called_shot)
decides BEFORE compose, so a lane one of them silences produces no take, no judge
score and no post at all. lane_dark and surface_dark both read judge scores, so
a lane held to zero by its own gate looks exactly like a slow chart week to both of
them. This finding reads the gate’s own decisions instead: agg.debut_gates rates the
SHIP reasons (_DEBUT_GATE_SHIPS — the one source of truth for that split) against
every decision, and flags medium when a key took GATE_DARK_MIN_DECISIONS (6)
decisions and shipped nothing. Keyed by <event>/<chart>, not by event, because
apple_debut is TWO lanes under one kind (the songs chart and the albums chart) and an
event-level rate stays green while one of them is dark — a real shape after #2207,
since the two charts carry different NEW rows and the depth floor can silence one
alone. A lane with no candidates emits nothing and is correctly invisible; one ship in
the window clears it. charted decisions are excluded from the rate entirely
(_DEBUT_GATE_NON_CANDIDATES) — the row was already on the chart, so there was no
candidate, which is the same state as a chart with no NEW rows. That exclusion is
what makes the finding usable, and it was found by replaying real events rather than
reasoned out: over 12h on 2026-08-09 the Spotify Global WEEKLY lane took 132
decisions and shipped none, all 132 charted, because the weekly chart only
refreshes weekly. Counting them, this would have fired on that lane about six runs
in seven. With the exclusion the same window produces ZERO findings across all eight
live lanes. Two more refusals are non-candidates since #4399 (2026-10-17). One
helper, _debut_gate_non_candidate, holds the whole rule:
called_shot stale OLDER than the lane’s own limit (_CALLED_SHOT_MAX_AGE_DAYS,
28 by default). The HITS projection has no last-week position for a catalog
re-entry, so the lane dates every such album and refuses it. Over 17 days that was
4,729 refusals, all 224 days old or more. Since #4445 the lane puts the limit it
applied on each row as detail.max_age_days, and the monitor and the panel compare
age_days to that. The monitor runs on a GitHub runner, so a copy of the limit read
there would not see a Railway override of MUSIC_DESK_CALLED_SHOT_MAX_AGE_DAYS. A
row emitted before #4445 has no limit, and _CALLED_SHOT_LEGACY_MAX_AGE_DAYS (28)
applies to it. A test pins that value to the cog’s default. A stale
YOUNGER than the window (a future release date) and every no_date still count.watch_new_entry deep. The lane recognized a re-entry and declined it by
position (MUSIC_DESK_WATCH_REENTRY_MAX_POS, or MUSIC_DESK_WATCH_RADIO_ENTRY_MAX_POS
on radio). On apple_debut, deep is a real debut the depth floor cut, so it still
counts there.Also, watch_new_entry radio_entry now counts as a SHIP: the lane posts it. The
desk_debut_ship_rate panel APL is generated from the same rules
(axiom_setup._debut_gate_ship_rate_apl), so the panel and the finding agree.
Measured by replaying the 14 days to 2026-10-17 in 12h windows (27,924 decisions):
gate_dark findings went from 120 to 68. called_shot dropped from 20 to 5,
watch_new_entry/it_songs from 11 to 3, and watch_new_entry/radio_us, sp_gweek,
it_albums, ap_songs and yt_us went to 0. What remains is real no_date: the 5
called_shot windows are one album the lookup cannot date (Ed Sheeran “+ - = ÷ x
TOUR COLLECTION”, 409 no_date), it_songs is several undatable songs (Fuerza Regida
“Suzuki” leads with 96), and ap_albums is LUCKI “Last Laugh (archive tape)” (75). The apple_debut and spotify_debut
findings (59) are unchanged: their terminal stale event carries no per-row age, so
the monitor cannot tell a catalog refusal from a real one there.
radio_debut is deliberately excluded: its three reasons pick
the FRAMING of a post that ships either way (#2077), so it has no rejection to go dark
on and would flag permanently. Read the desk_debut_ship_rate panel to see the split
and desk_debut_gates for the reason mix behind it.
tier_gate_off, #2214; reads
spotify_debut as well as apple_debut since #4175, because both stamp tiers) — the #2207
floor asks aw.recognition_tier which bar an artist must clear, and an EMPTY
recognition list resolves every credit to known, relaxing the floor from the unknown
bar to the looser one. That is the correct fail direction on a kworb outage and it is
completely silent: the lane keeps posting, nothing raises, and the posts are still
well-formed — they are simply held to a bar the owner did not set. So every
apple_debut event stamps tiers (live / off) and ANY off decision raises this
medium finding; there is no rate floor, because one window of a loosened gate is
already worth naming. The cause is either the MusicDesk cog not being loaded or
_known_artists() failing its build (a partial build is forced empty on purpose), so
look for an artist_watch error beside it.*_scored is not always a JUDGE score — some surfaces emit the same event as a
telemetry MARKER with a hard-coded score=0.0: a per-slot compose budget hit
(reason=attempt_cap), a degradation notice (reason=watchlist_unavailable), or a
deterministic guard dropping a line before the judge ever saw it (the whole
phase=shape_reject), or the SALVAGE marker (reason=ungrounded_salvaged, #2571) the
music-desk reveal lane emits when it re-asks and ships a clean retry — that one even
carries shipped=True and a real lane in phase (“reveal”), so it must be excluded by
reason, not phase. ops_monitor.is_judge_score excludes them all from quality stats,
where they would otherwise read as the judge scoring a post badly and inflate the
below-floor rate. Measured on 14 days of real music-desk rows the pollution was tiny
(1 marker in 1,840), so this corrects an inaccuracy rather than a live false alarm —
but watchlist_unavailable is new and attempt_cap scales with lane count.breaking_lane_stale) — the breaking lane’s whole value is
being FIRST (owner steer: engagement follows the break, not the recap). The lane orderer
hard-caps eligibility (BREAKING_WINDOW_HOURS, 1h since the 2h→1h tightening landed on
measured supply — p50 41–63m across the wire desks), so a p50 source age over 45m
(BREAK_LANE_P50_CEILING_SECS, min BREAK_LANE_MIN_POSTS posts) on any desk’s
breaking-lane posts means it consistently ships near-stale breaks — slots too sparse
(firehose off?) or the wire fetch lagging — a drift no error path surfaces. Flagged
medium per desk off the *_posted events’ folded lane/source_age_seconds (the
lane-scoped subset of the render-only “Break latency” baseline table). The off-slot
breaking EVENT lane (cogs/event_poster.py) is the cadence fix this finding was waiting
on: fast posts carry a folded fast=true on the same *_posted events (query
parse_json(detail)['fast'] to split fast vs slot breaks), and because they ride the
same lane/source_age_seconds fields they pull the measured p50 down with no monitor
change — this finding doubles as the fast path’s silent-death watchdog (a dead fast
path drifts the p50 back toward the ~40m slot-cadence baseline and re-fires it).chart_capture_stale, #3562 A) — nothing upstream archives the
day-one platform charts (Spotify US daily, Apple Music songs + albums, iTunes songs +
albums), so a day the daily walk misses is lost for good. The walk emits one
chart_history phase=freshness row per 6-hour slot with each chart’s lag in days in
context. The aggregate keeps every freshness row and reads the NEWEST one: a chart
past its limit (chart_history.FRESHNESS_LIMITS: 3 days for the Spotify page, which
kworb dates a day or two late; 1 day for the rank charts; a chart never stored counts
too) files medium, naming each chart and its lag. A window with at least
CHART_HISTORY_DARK_MIN_EVENTS (500) events and NO chart_history row at all files
high: the job’s tick loop has stopped. Skipped when the window was truncated. The
per-source ok rate (chart_history:daily) cannot see either case, because a walk that
does not run emits nothing. Panel ch_fresh.music_lag_slow, #3829) — the owner target (epic #3821) is a music
number in the room within 10 minutes of the upstream change. The aggregate keys each
music post’s folded source_age_seconds by <surface>/<lane> (the folded lane on the
newsroom, else phase). It counts music_alert_posted and music_desk_posted, and only
the watcher lanes of music_news_posted (MUSIC_LAG_NEWS_LANES: x_watch and, since
#3834, feed_watch, the two sources of the newsroom’s EventPoster lane; the newsroom’s
other lanes are recaps of hours-old tweets by design). Both watcher lanes also feed the
breaking_lane_stale series. A staged audition (delivered=staged) is left out: it
reached #bot-logs, not users. A lane whose p90 is over 600s
(MUSIC_LAG_P90_CEILING_SECS) over at least MUSIC_LAG_MIN_POSTS (5) posts is flagged
medium. A slow lane raises no error, so nothing else shows it. The report also
renders a “Music lag” table with every lane, the ones under the post floor too. It is
not a LATENCY_CEILINGS_MS entry: that table gates duration_ms, the time of one call,
and this is the age of the data at post time. The baseline before this landed (7 days to
2026-10-05, rebuilt from the DB, Kalshi and the HITS dedup stamps): alert settle p50 4m /
max 9m (n=3); alert crown lanes p50 13s to 12m (n=1 to 5 each); desk chart_proj p50 7m /
p90 2h 56m (n=33); desk reveal p50 36h / p90 69h (n=28, it posts on slots).
music_desk/reveal is recorded and rendered but NOT gated (#3842,
MUSIC_LAG_UNGATED_LANES). It is the desk’s slot recap of a Kalshi settle. The owner
ruling on #3821 gives settles one fast poster, the music_alert settle lane, so the
settle’s speed is gated on music_alert/settle. Gating the recap would fire on every
run. The report marks it “(recap, not gated)”. music_desk/chart_proj after #3842:
the lane’s 3-board cap used to end the watcher pass with the rest of a new HITS
document’s boards unposted, and they waited for an unrelated source change or a slot.
Now the pass reports itself not complete, so the next look posts them. Replayed on the
documents the watcher handled (2026-09-29 to 10-02), p90 17m -> 11m (p50 6m); the two
older documents (p90 2h 56m) predate the watcher. A deferral emits source_change
reason=deferred with detail.sources=["deferred:chart_proj"], so it shows as its own
row on the source_changes panel.settle_not_posted_fast, #3846) — music_lag_slow reads only the
settles that POSTED. A settle that music_alert composed and then dropped has no age, so
no lag row shows it. This finding counts every music_alert settle in the window (the
pure settle_coverage over Aggregate.settle_rows) and sorts it three ways:
music_alert_posted phase=settle with source_age_seconds of at most
600s (SETTLE_POST_WINDOW_SECS);source_age_seconds. Before SETTLE_AGE_SINCE
(2026-10-05 02:00Z, after #3839 deployed) this is history and only counts. After it,
the lag field broke and both latency findings would read green, so it flags;topic_dedup verdict when music_alert posted a settle
of the same album (same subject, or the same detail.ticker when both rows carry
one) within SETTLE_SIBLING_SECS (3 days) of it, in either order. One album has up
to three settle markets in one weekly batch, and the judge drops a metric that a
sibling post already covered. These are intended, so they do not flag; the finding
names their count. Only a topic_dedup drop can be covered: a judge, rule or shape
drop stays a miss. A desk or newsroom post that states the figure is NOT joined (the
rows carry no figure to compare), so a desk-first repeat still counts as a miss.
Measured on the 14 days to 2026-10-06: before, 25 of 39 not posted; after, 8 of 39
(17 covered). The 8 are the 7 real misses of the #3846 classification plus the
correct desk-first repeat (Showgirl Encore units);topic_dedup verdict that no sibling post covers, or a terminal
#3822 marker (settle_rejected, settle_retry_cap). A topic_dedup row from after
#3902 names its subject in matchup. An older row has none, so it takes the subject
of the guild’s last shipped settle score row. A settle_retry or settle_capped
(#3889 gap 3, in NON_SCORE_REASONS; the cog does not emit it yet) marker is not
terminal and is skipped. A marker is named by the last drop row for its subject: a settle shape guard
(a “settle: “ matchup, or release_title / attribution_tag, which run only on a
settled reading; metric_word:* runs on every signal with the bare subject, so it is
left out) or an ungrounded_* rule
(SETTLE_RULE_REASONS) names itself; any other phase=settle drop is the 0.6 judge,
named score_below_floor whatever its score (the rubric allows 0.0) or wording. A marker
within SETTLE_PAIR_SECS (60s) after a topic_dedup miss on the same subject is that
settle and is not counted twice; a later marker on the same subject is another market
of the album and counts. The rows of one settle often share a millisecond and Axiom
returns those in either order, so settle_coverage sorts on (ts, emit rank: drop row,
then topic verdict or post, then marker) before it joins them. Rows key on the
first 72 characters of the subject, because ship_guard cuts “settle: competitor_slow, #3862) — the owner goal (epic #3821) is
the fastest chart account. competitor_section runs scripts/competitor_scoreboard.py
each ops-monitor run: it scores our music posts from 24h to 12h before the run hour
against @chartdata / @talkofthecharts / @chartmastersorg / @hitsdd (one twitterapi.io
search, TWITTERIO_KEY read from the Railway variables), ingests competitor_match
rows, then reads the last 7 days. A lane (<surface>:<phase>) whose 7-day win rate is
under 50% (WIN_FLOOR) over at least MIN_MATCHES (10) pairs is flagged medium.
A lane that loses races raises no error, so nothing else shows it. The report always
carries a “Competitor scoreboard” section with the run line and the 7-day table; a
failed run or a failed import says so there, so a dead scoreboard is visible in the
report and as a missing kind=run row. utils.music_news needs aiohttp, so the
job installs it in its own step (#4159). Before that step, the section read
“Skipped” on every run (#3862).scorer_truncated) — a *_score Haiku judge whose JSON verdict
hits max_tokens gets cut off (the #697 root cause: a truncated reason that dropped the
score to 0.0). Tracked off the claude_api stop_reason; a sustained truncation rate
(SCORER_TRUNC_RATE 15%+ of SCORER_TRUNC_MIN+ calls) on any judge is flagged high
with “raise the budget” — the cause-specific signal that points straight at the bug,
vs surface_dark’s cause-agnostic downstream symptom. The *_pick/*_confirm
selector judges are watched by the same finding (#1935: subject_image_pick truncated
at its old 60-token budget mid-narration, before emitting any {"pick": N} verdict,
and the digit-scraping parse fallback turned “Photo 1: …” into pick=1 — the wrong
person shipped to X as Tarik Skubal. The parse now refuses prose, so a truncating pick
judge LOSES its picks — this finding is what surfaces that loss).reasoning_exhausted) — a turn that spent its whole output
budget on thinking and returned NO text (#2867 on Claude, #2869 on GPT). It reports
ok=true — reasoning exhaustion is not an API error — so neither the failure rate nor
error triage can see it; the surface just ships its “lost my train of thought” fallback.
Both providers fold into one table keyed like the latency map (a bare purpose for
Anthropic, openai:<purpose> for the GPT path), read off detail.thinking_exhausted /
detail.reasoning_exhausted and their *_recovery twins. The FIRST exhausted turn is
not a finding: the one retry usually absorbs it. What flags (medium, at
REASONING_MISS_MIN = 1, because it is meant to be rare) is a turn stamped BOTH
exhausted and recovery — the retry came back empty too, so an answer was lost. The fix
is that surface’s budget floor (_THINKING_MAX_TOKENS_FLOOR / _REASONING_TOOL_TOKEN_FLOOR),
never a wider retry. A “reasoning budget” line renders every run, including its
all-clear, so a zero rate is a stated result rather than silence. Counting note: a
recovery-only turn is deliberately NOT counted as a recovery, because the Anthropic
loop keeps that marker set for every later turn in the same call and counting those
would hide misses; recovered is derived from the two unambiguous counts instead.settlement_broken
(any bet_settle_tick ok=False → open bets aren’t paying out, high), settlement_stalled
(bets placed but ZERO settle heartbeats → the loop is dead, high), bet_place_failing
(system-reason placement fails — game_gone/no_line, the stale-slate race — over a
rate, excluding user-choice bad_stake/insufficient_funds), payout_undelivered
(settled-result pings not landing), picker_degraded (/bet opens served
loading/empty), and slate_stale (the bettable game LIST passed
SLATE_STALE_SECS = 30 min before it refreshed, read off bet_slate_refresh’s
detail.refresh_age_s, high). bet_slate_refresh also carries detail.dropped: the
games the market lock removed from the slate. A settled game counts for a few hours;
a count that stays high all day means the lock hides games that have not played (the
2026-10-21 preseason-rematch bug, where a date-less lock row hid four NBA games).
The settle loop emits bet_settle_tick EVERY tick (the
alive-heartbeat) and a paired error (source=bookie_settle) on failure, so a
silently-broken payout is caught by the next ops-monitor run.
slate_stale exists because every betting surface — the morning board, the pregame
line, /bet and BET PLACEMENT — reads ONE cached game list, and a list that stops
refreshing is silent: each surface keeps posting, the picker keeps serving games, and
the only trace is a refresh that stops happening, which reads exactly like a quiet
period. The warm loop’s cheap live-odds re-fold stamped the game list as freshly
fetched, one game stuck on live kept the re-fold running, and the list froze.
It could not detect that freeze RETROACTIVELY, and that limit is worth understanding
before trusting it (Codex review on #3228). The freeze stopped the refresh path from
being reached, so nothing emitted a growing age; and the startup refresh a redeploy
fires reports ~0, because _before_warm seeds the cache from the durable row moments
earlier. Measured on the real incident, refresh_age_s reads 0s in both a frozen and
a healthy window.
The obvious alternative was built, measured, and CUT. Correlating betting-surface
posts against an ABSENCE of demand-triggered refreshes (trigger=open/cold/place)
looked decisive on a narrow sample, and then fired on 13 of the 22 windows before the
freeze was thought to begin when replayed over 14 days. Demand refreshes are bursty
in normal operation, so their absence does not mean the list is frozen, and a
high-severity finding on more than half of all windows is alert fatigue on the one
interface the owner reads for QA. Designing a sound freeze detector is #3230.
What makes slate_stale sound going forward is the stamp fix itself. The reason
the age read 0 is that the re-fold stamped _bettable_cache without fetching, so the
list reported itself fresh and never aged. With the two clocks separated the stamp no
longer lies, a list that stops refreshing genuinely accumulates age, and the next
refresh reports it. The metric could not work retroactively because the bug it measures
is the bug that corrupted it.
The slate dashboard panel graphs refresh_age_s as max_age_s.
Since #4218 nothing refreshes the slate while nobody bets, so a list can sit
unread for hours. _bettable_games now DROPS a list older than _MAX_STALE_SECS
(15 min) and fetches cold (trigger=cold, refresh_age_s=0), because that list is
never served. So refresh_age_s still measures how old a list grew WHILE it was
served, and the first /bet open after an idle night does not file a false
slate_stale. The boot seed is stamped one TTL old (180s), so the first refresh
after a reboot reports about 180s.roster is the artist curator, its own surface since
2026-09-07, on the same kinds stamped surface=artist_curator – the image/text mode split) so the content curator is
eyeballed even when fine — its surface-dark watch, since the fit-judge is pick-or-skip
with no 0-1 score to grade. Finding: curator_dark when she posted ~nothing over
CURATOR_DARK_MIN+ slot attempts (none_fit-dominated → high, users see silence; a
supply issue no_seeds/no_history → medium, the channels just need links). Reads the
curator_posted + curator_evaluated funnel; curator_fetch integration health (the
twitterapi.io crater, self-healed by the health cog #1257) backs it up.x_game_round: x_game_open_failing (high)
when opens fail at/over X_GAME_OPEN_FAIL_RATE across X_GAME_OPEN_FAIL_MIN+ attempts —
the game can’t post a round (dominant reason named: no_song=pool-gen/iTunes down,
upload_failed/post_failed=X write, unprovisioned=env vars); and x_game_no_wins
(medium) when X_GAME_NOWIN_MIN+ production rounds close with ZERO wins — clips post but
no guess ever lands, so the answer webhook (the twitterapi.io filter rule + /x/webhook,
x_filter_rule/x_webhook events) likely isn’t delivering, or the pool is too hard /
no engaged followers yet. Staged (audition) rounds are kept OUT of the win funnel so a
staging preview can’t false-trip the no-win watch.card_art integration health, #youtube-chart-art) — every rung of the
market/betting card’s background-image ladder is fail-open, so an exhausted ladder
raises nothing: the card ships the branded floor and error-triage never sees it. The
owner did — “still needed art”, a Kalshi music-video market carded bare to Discord AND
to X. So market_subject now feeds the observe-everything OkRate (card_art), with
cache hits excluded so the rate measures resolutions actually attempted. ok=False is
a LEGITIMATE outcome for a market with no imageable subject, so the steady rate is low
by design (12 of ~700 cards over 30 days) and only a real blackout reaches the 0.6
integration_unhealthy flag. The source field names the rung that answered
(catalog / sports_api / youtube_chart / found / market_leg / native /
none), so one rung dying is visible in the mix even while the total rate holds.
The blind spot this rate does NOT cover: a lane that never CALLS the resolver.
card_art grades resolution QUALITY, and it can only grade the attempts it sees — a
card built with no art call emits no market_subject at all, so it does not lower the
rate, it is simply absent from it. Both art-less lanes found so far were invisible this
way (the DTF St. Louis floor, then the RT Final card + the X-mentions projection card,
2026-08-31), and in each the X media gate compounded it by PASSING the bare number card
as coherent — correctly, since the take matched the number. So the diagnostic for
“this card shipped with no art” is to check whether a market_subject / market_image
event exists for the post at all: absent means the art was never resolved (a code path
that skips the ladder), present with source=none means the ladder ran and came up
empty (a resolver problem). The two have different fixes and look identical in a
screenshot. Counting art COVERAGE per card-shipping surface would close this properly;
it is not built yet (tracked as a follow-up).media_ungated, owner steer 2026-09-16 – the X media audit). Every photo that reaches X is meant to pass x_crosspost._media_gate at the one chokepoint (maybe_crosspost; crosspost_with_quote is a thin wrapper over it), so per surface the PHOTO checks (media_coherence with no detail.kind=clip) must be at least the media tweets (crosspost delivered=posted mode in (media, number_fallback)): a multi-photo tweet gates each photo, and a number_fallback tweet exists only because a check dropped its photo. A surface whose tweets exceed its checks has a path shipping media unjudged. HIGH when the gap is two or more; a gap of exactly one can be the window edge (a check at 11:59, its tweet at 12:00) and reads medium – verify, never dismiss. Measured over the 30 days to 2026-09-16 the shape held on all 13 surfaces (music_desk 1534 checks / 1528 tweets, music_news 567 / 567, market_alert 294 / 293, bet_board 56 / 56, x_mentions 28 / 28 …). What it does NOT cover, by design: x_game, which posts through XPoster directly and emits x_game_round rather than crosspost – its media is a clip it renders itself plus the catalog row’s own artwork by id, never a scraped image, and a future lane that copies that shape must either route through maybe_crosspost or be added here. The health line reports tweets vs photos judged every run so the balance is watchable when it is healthy too.media_copy_mismatch, the “art failure” report
2026-08-31) — the delivery media gate read the image as the RIGHT subject and the
take’s WORDING or NUMBERS as disagreeing with what the card displays (“the graphic says
5.3B but the post copy says 5.2B”). The image ships, so this is a COMPOSE bug, not wrong
art, and it is counted apart from the wrong-art rate — mixing the two is what hid it for
30 days while it deleted correct chart cards. Medium: a wrong figure reached the
timeline, but the card beside it carries the right one. Read the card against the take;
the card is rendered from the parsed rows, so where they differ the CARD is normally
right. Since #4367 COPY means a value the image DISPLAYS that contradicts the take.
It used to also cover “a rank the chart does not show, a shortened or missing credit”,
so most of the 279 flags in the 30 days to 2026-10-17 were takes that ADDED a fact the
card does not draw (“up from No. 6” over a #5 card). Re-run on 332 real shipped posts,
the new definition cut COPY from about 42% to 11% of the flagged posts and from about 7%
to 1% of unflagged ones. The rest is still noisy: of 21 posts it flagged, about 6 were
real contradictions and about 14 were the judge miscounting a card’s rows. That is why
COPY still never drops the image or the post. Expect this count to fall to about a
quarter of its old level; a count that stays high is a real compose or card bug.(source, exception_class), split by
the recoverable tag from emit_error (non-recoverable = high / user-impacting;
all-recoverable = low, bumped to medium on a burst ≥ BURST_ERROR_MIN), with the
inline traceback + context attached. Forwarded ERROR/CRITICAL logs are triaged too
(#ops-sees-logs) — they weren’t, and that was the report’s largest blind spot: a
surface that only log.exceptions was invisible here no matter how loudly it failed.
The music-alert seed overflow raised ~2.2K ERROR lines a DAY for days, every one
shipped to Axiom, none of them reachable by this report, while the affected markets sat
permanently dark. Two things had to change: fetch_error_logs_axiom (the main event
query filters event != 'log', so the rows never arrived — it is a SEPARATE bounded
query, not a relaxed filter, so a log-storming surface can’t evict the metric events
the rest of the report is built from), and a grouping key, since a log line has no
exception class — _log_error_signature takes the message before the first : with
long digit runs masked, so one recurring failure stays one finding instead of shattering
into per-id singletons and slipping under the burst test. Such a signature carries no
recoverable flag, so it counts as unknown and can reach medium by bursting but
never high — folding logs in can’t outrank a real non-recoverable error. Log triage is
Axiom-only; the Railway fallback reconstructs events from EVENT stdout lines, and a
plain log line isn’t one. Verified against a live 12h window: it surfaces the seed
overflow as 1152x music alert seed failed at MEDIUM with the asyncpg traceback
attached, plus four other previously-invisible signatures (a MenuView item exception
among them), growing the report by 5 findings — signal, not flood.The report also renders an unconditional Quality spot-check (recent posts + their self-gate scores per surface) so the judge eyeballs voice/quality even on an otherwise clean run — this is what makes it a quality eval, not just a regression alarm.
claude-code-action then judges the flagged samples and files deduped auto-eval
issues (it searches open auto-eval issues first, so no spam; “All clear” files
nothing). The deterministic core (parse/aggregate/evaluate/render) is pure and
unit-tested in tests/test_ops_monitor.py; the Railway I/O is integration-only
(pragma: no cover).
This replaces the standalone Railway log-monitor routine — error monitoring now lives here, one interface and one cadence. There is no separate log-monitor workflow.
deploy_failed / deploy_stale — the change that never landed (#2088)The only ops-monitor finding that does NOT read the event stream, because the question it
asks cannot be answered from there. Every other blackout check here reads what the bot
EMITTED, so all of them assume the bot is running the code we think it is. A failed deploy
breaks that assumption in the one way nothing else can see: main carries the change,
Railway keeps serving the PREVIOUS build, and because that build works fine every surface
looks healthy — no dark lane, no error, no latency blip. The change simply never happened
and nothing said so. (_recent_deployment_ids was already asking Railway for status and
discarding it.)
fetch_deploy_findings asks Railway what is actually running and compares it to the commit
Actions checked out (GITHUB_SHA, which on the schedule is the default branch’s tip).
A local run has no GITHUB_SHA, so it compares against origin/main (main_ref, #4175).
It used git rev-parse HEAD until #4175, and a run from a feature branch then compared
production against that branch and raised a false deploy_stale. Run git fetch origin
main first, so origin/main is current. The commit age is read off the same commit. The
pure core is deploy_health and is unit-tested.
| finding | fires when | why it is separate |
|---|---|---|
deploy_failed (high) |
the NEWEST deployment has status FAILED | names the stage — no image digest means it died in the BUILD, which is a different fix from a crash or a failed healthcheck |
deploy_stale (high) |
the live SUCCESS deployment’s commit is not main’s tip (GITHUB_SHA, else origin/main), and that commit is older than DEPLOY_STALE_AFTER_MIN (45m) |
catches a build failure Railway has already tidied away — the FAILED row can be gone while the old SUCCESS row still serves, so status alone reads clean. Only the COMMIT comparison sees it |
Both fail OPEN: no Railway creds, a GraphQL error, or a git command that will not answer all return no findings. A deploy check that broke the ops run would be worse than the gap it closes. The 45m allowance exists because Railway builds in a few minutes — flagging a commit that landed seconds ago would make the check cry wolf twice a day.
Verified against live Railway both ways: silent when production served 005acd7 and main
was 005acd7; fired with the correct detail when handed a main SHA production had not
reached.
cover_fuzzy_high / cover_overrule_high — the cover exemption’s two health signals (#3520 / #3530)The delivery media gate trusts a catalog_cover over the vision judge. That trust is earned per row since #3530: the row’s credit must verify exactly (apple_music.credit_verified, no act the request never named) and its core title must equal the request (apple_music.title_verified). A cover the picker resolved but could not verify that way reports catalog_cover_fuzzy and is judged by face. Two findings read that split:
cover_fuzzy_high (medium, per desk) — the fuzzy SHARE of the catalog-cover family (_COVER_RUNGS) is at or past COVER_FUZZY_RATE (a third) over at least COVER_FUZZY_MIN (8) covers. A small share is normal (a stylized spelling, a remix the post itself names). A run of plain titles means the exemption rule is too strict or a source changed how it spells acts. It is silent otherwise: every such card still ships a picture. Read the desk_art rows with source=catalog_cover_fuzzy beside their market_card_hero to see the subjects. Dashboard: the cover_exemption panel on the Desks board.cover_overrule_high (medium) — the exemption kept at least COVER_OVERRULE_MIN (3) covers the judge flagged as the wrong SUBJECT in one window (media_coherence reason=verified_source, detail.source=catalog_cover, detail.verdict=subject; a portrait overrule on deezer_artist_exact never counts, because there the judge is the one that is wrong, five out of five in the 30-day log). After #3520/#3530 each one should be a real cover that does not show the artist (Charli xcx’s “Music, Fashion, Film” is three men in a kitchen). The samples carry the judge’s notes: a note that NAMES a different artist, a remix or another release is a wrong pick that shipped, the class those PRs closed. The Axiom monitor “Tootsies: cover exemption overruling the judge” fires on the same count past 2 in 12h. Baseline before the fix: about 19 in 30 days. Since the text-evidence question (2026-09-20), a note that names a different act in PRINT no longer reaches this finding: that flag drops as reason=mismatch with ` |
text: in the note. What is left under verified_source on catalog_cover is the face-only misread, which is the class the exemption exists for, so a sample here that NAMES another artist means the text question answered SAME on readable text -- read the claude_api purpose=media_text_evidence` row beside it. |
scope_unnamed — the newsroom’s chart list narrowing on us (#2127)The newsroom used to relay whatever chart a trusted account posted about. It was the only
chart surface with no chart list: the movement lane reads MUSIC_DESK_CHART_LANE, the
boards read MUSIC_DESK_STANDINGS_CHARTS, the genre boards read billboard.GENRE_CHARTS,
and the newsroom read the ACCOUNT and the number cue only. Over the 30 days to 2026-08-07
its 135 chart stories named about 20 chart families, including ARIA, the U.K. Official
Albums Chart, Billboard Japan, Bubbling Under, Kid Albums, Top Movie Songs and a Dabeme
fan tally of “Top 100 Male K-Pop Vocalists”. The owner reviewed that list and cut most of
it; music_news.KEPT_PATTERNS / CUT_PATTERNS is the reviewed scope, applied by
_chart_in_scope right after classify so an out-of-scope story costs no verify and no
compose.
What this finding watches is NOT the cut. A drop that names its family
(family=uk_official) is the owner’s decision working, and it never raises anything,
however often it fires. The gate fails CLOSED, so it also drops a story naming no chart it
recognizes, as family=unnamed — and that bucket is a different thing entirely: the
patterns stopped matching a chart we meant to KEEP.
That failure is invisible to every other check here. It raises no exception. It fails no
integration, so no OkRate moves. The story dies at the inbound edge, before the compose
and before the self-gate, so surface_dark and low_quality — which read judge scores —
see a normal week with slightly fewer posts. The desk simply gets quieter, and the quieter
it gets the more it looks like a slow news week.
The finding fires at medium when unnamed is at least CHART_SCOPE_UNNAMED_RATE
(25%) of at least CHART_SCOPE_MIN_DROPS (4) scope drops in the window. The floor is far
above the measured baseline: 1 of 135 stories before the gate shipped, and that one was an
artifact of the 120-character preview the measurement read (the live gate reads the claim
plus the whole source post, so it names the chart more often). The minimum count is low on
purpose — the gate drops roughly one story a day, so a higher bar would never fire.
The fix is a row in KEPT_PATTERNS, not a threshold change. Re-measure against real posts
with python -m scripts.dryrun_chart_filter (it reads the last 30 days of shipped chart
stories out of Axiom and prints kept / cut / unnamed with the unnamed lines in full).
Dashboard panel: desk_chart_scope on the Content Desks board.
discord.py sends a heartbeat to the gateway from the event loop. When sync code holds
the loop for 10s or more, discord.py logs heartbeat blocked for more than 10 seconds
with a traceback of the loop thread. Discord then often drops the session: the log shows
Attempting a reconnect, then RESUMED. While the loop is blocked, the bot answers
nothing, so users see lag.
These lines are forwarded log events (logger discord.gateway / discord.client),
not EVENT lines. Nothing read them until 2026-09-28. Over the 30 days before that date:
| Signal | Count |
|---|---|
heartbeat blocked |
157 |
Attempting a reconnect |
37 |
websocket is Ns behind |
4 |
| fresh logins (each one a deploy, checked against Railway) | 463 |
The blocking frames were almost all two causes: Pillow card renders on the loop
(render_number_card / render_board_portrait through build_market_card,
build_number_card and build_board_card), and artist_watch.credited_names
rebuilding its name index for every RIAA row. The fix moved the renders to
asyncio.to_thread and caches the credit index per list pair.
The RIAA board came back on 2026-10-12 and 2026-10-16 (16 warnings in 7 days, #4291).
Every traceback sat in RiaaClient.awards_for_act on the roster walk. Each act’s read is
a cache hit, and an L1 hit never yields, so the walk parsed about 60 MB of cached pages
and ran ~490k fold_tokens calls as one stretch on the loop. The fix parses every RIAA
page read in asyncio.to_thread (riaa._off_loop) and memoizes fold_tokens. On the
real cached pages the longest stretch went from 5.6s to 0.04s (local CPU).
gateway on the Health & Ops board. It counts blocked,
behind, reconnect, resumed and login lines over time. A resumed with no
blocked before it is a normal reconnect that Discord asks for.loop_stall (utils/loop_watch.py) names the code behind every
1s+ stall, which discord.py does not report. Panel loop_stalls.Tootsies: event loop blocked (Discord heartbeat) fires on 3+ blocked
warnings in 6h. To triage, read the warning’s traceback in Axiom and find the deepest
/app/ frame. That frame is the sync work to move off the loop.['tootsies'] | where event == 'log' and logger == 'discord.gateway'
and message contains 'heartbeat blocked' | sort by _time desc