tootsies

a discord bot for the tootsies server. ask, recap, discuss, ship features by typing.

Project maintained by mejasonmejason Hosted on GitHub Pages — Theme by mattgraham

Engineering rules register

The extensible home for engineering rules — curated from the strongest public practice (Anthropic’s agentic-coding guidance, the Karpathy CLAUDE.md line + community extensions, Sentry’s and OpenAI’s agent guides, Cloudflare’s template standards, HumanLayer) and adapted to this repo. Skim at the start of any non-trivial change. CLAUDE.md’s always-loaded judgment section stays tiny; rules land HERE.

The register’s constitution. A rule needs a provenance tag plus either a local incident or strong multi-source consensus; personality rules (“act like a senior engineer”) are banned as measured no-ops; a rule that measures as a wash gets cut (the docs/PROMPT_OPTIMIZATION.md discipline, applied to rules about ourselves); and promotion into CLAUDE.md usually trades against something coming down — the always-loaded file does not re-grow. Rules are kept to their load-bearing form: if a sentence only restates its rule, it goes.

Scope and diffs

Surgical changes — clean up only your own mess. [Karpathy] Touch only what the task needs: no drive-by reformatting, renames, or refactors of adjacent code. Cleanup is scoped the same way — remove only what YOUR change orphaned; pre-existing dead code gets mentioned (PR body / an issue), never silently deleted.

Ship big refactors in reviewable stages. [community] A multi-step migration lands as a sequence of individually green, individually revertible PRs — mechanical rename, then behavior change, then cleanup. A stage that can’t go green alone is a design smell in the split, not a reason to merge red.

Deciding and assuming

State assumptions; verify the cheap ones. [Karpathy] A load-bearing assumption gets verified (most are one query/grep away here) or stated where the reviewer will see it. “Assumed X, unverified” is fine; silence is not.

Surface conflicts — never average them. [community] When two parts of the codebase, a doc and the code, or two owner steers disagree, name the conflict and pick a side explicitly — owner for judgment, evidence for fact. Blending produces code that half-follows both and launders a decision nobody made.

Time-box the rabbit hole. [community] Past ~2× the effort you’d have estimated, stop digging: restate the problem, switch approaches or surface it. Sunk effort is not evidence the approach is right (#567: the step-back found a 10-line answer to a 300-line grind).

The second patch to the same abstraction is a redesign checkpoint. [OpenAI] Adding a second condition/branch/special case to something you already patched once means stop: re-derive the abstraction from the full case set — one redesign beats accreting compatibility branches (#824: banning crutch phrases one at a time never converged; the self-dedup redesign did).

A significant behavior change states its scope contract. [OpenAI] In the PR/issue: the required behavior, plus what is INTENTIONALLY unsupported and how it fails (skip? fallback? raise?). The unstated-unsupported case is where the next incident lives. (docs/INTEGRATIONS.md’s omitted-and-why column, applied at the behavior level.)

The deterministic / model-judged boundary

Repeatable behavior lives in code; model judgment is for taste. [OpenAI ext.; this repo’s architecture] Retries, routing, thresholds, caps, settlement math, dedup keys — same inputs, same outcome — are deterministic code with tests; the model judges language, relevance, quality. A number or branch decision living inside a prompt is a bug of this class. The mirror holds too: don’t hand-code brittle heuristics for a genuine judgment call (the classifier-vs-regex history). The repo-wide inventory of every keyword list, string match and regex that makes a decision, with a per-site verdict and the ranked replacement worklist, is docs/CLASSIFIER_AUDIT.md (2026-09-18, refreshed 2026-10-18).

A regex matches only a closed set; a model decides an open set. [owner, 2026-10-18; stronger wording adopted 2026-10-19] Three rules:

  1. Code only for a closed set, checked by an exact match. A closed set is a fixed format (a number, a date, a URL, a ticker, a key), our own enum or payload field, or a structural parse. A partial or fuzzy match that stands in for meaning is not an exact match. An open set stays code only in three cases: a safety floor (rule 3), the protected classes listed below, or MEASURED-KEEP (below), which covers a site where a model was measured on real inputs and was worse, and a site with no real traffic.
  2. A model decides meaning, and the list is the model’s input. Outside those cases, Haiku, Sonnet or Jev decides every open set of natural language when the model answers: a word, phrase or name list, a cue list, or a regex that approximates meaning. Haiku (ClaudeClient.classify) is the default for a conversion; a conversion uses Sonnet only when Haiku measures worse on the same inputs. A classifier that already runs on Sonnet for a stated reason (for example ClaudeClient.classify_mention_intent, where a misroute is costly) keeps it; moving it to Haiku needs the same measurement first. Jev (utils/typesafe_client.py) is not a Haiku replacement (owner, 2026-10-17): it is used only where latency or call volume earns it (Haiku is too slow, or one Haiku call per question makes too many calls) AND the answer fits a typed Noul, Choice or Score. When the model errors, the old rule decides as the fallback (MODEL-DONE). That fallback is part of rule 2, not an exception to it.
  3. A safety floor may stay as code, with a model on top. A floor is a code check that only blocks: it keeps a banned or false post out. It never approves a post on its own. A model reads what the floor lets through. A floor is recorded as PROTECTED in the registry (KEEP-PROTECTED in the audit), and its reason names the ban or the false claim it blocks. A list that only filters for relevance or quality is not a floor.

The old list is not deleted. It becomes input to the model, as reference examples or as the closed label set, and the old rule is the fallback when the model errors. The surface keeps its fail direction. “Hot path” and “a model judge runs after it” are not reasons to keep a word list: Jev answers in about 79 ms. Three classes stay as code until the owner signs off: settlement and money math, the constitution and the memory fence, and the permission allowlist.

A new regex or word list must be registered, and it may not add debt. [this repo: wave 19, owner 2026-10-18] tests/test_classifier_registry.py enforces the rule above. It scans every MODULE-LEVEL re.compile(...) and every module-level frozenset / set / tuple / list / dict with 3 or more string literals in cogs/, utils/, the repo-root modules and the runtime scripts. Each site needs a line in tests/classifier_registry.txt: path:NAME VERDICT reason. The verdicts are CLOSED-SET, PROTECTED (a deliberate fence: constitution, safety, money, privacy), MODEL-DONE (a model decides; the code is only the model-error fallback or the model’s input; the reason starts purpose=<name> or jev=<name>, and the test fails when that purpose is not a literal in the code), MEASURED-KEEP (an open set that stays code because a model was measured on real inputs and was worse, or because the site has no real traffic (#4522); not debt; the reason starts #<issue or PR> measured: and states the result or the 30-day count; re-measure when a better model ships or the count grows), MODEL-FALLBACK (an open set the code still decides), UNCLASSIFIED (the 2026-10-18 baseline) and OWNED-PENDING (an owner in _PENDING_OWNERS is converting it now; only that owner’s PRs touch those lines). The count of MODEL-FALLBACK plus UNCLASSIFIED lines is a ratchet: it must equal _DEBT_CEILING, which may only go down. A new site must be CLOSED-SET, PROTECTED, MODEL-DONE or MEASURED-KEEP. A conversion PR moves its line from MODEL-FALLBACK to MODEL-DONE and lowers the ceiling by one. A conversion or a move passes with no registry edit: a line with no site prints a note and does not fail, and a site moved to another file matches its old line by NAME. python tests/test_classifier_registry.py prints the missing lines. It runs in lint-typecheck (it is in LINT_STYLE_TESTS). The scan does not see inline patterns in a function body or class attributes, so a reviewer still reads those.

A model’s DECLINE has to be recognised in every shape it arrives in, because a missed decline is published as content. [this repo: 2026-09-06] Every compose surface skips by returning the single word EMPTY, and is_empty_response decides. It matched a LEADING sentinel only, so a decline that put the reasoning first and the sentinel last read as a real post and shipped: four went to the live rooms over two days, each one the model’s private reasoning about why it was declining (“…same story as the last read.\n
nEMPTY”). The self-gate cannot save you here – it scored them 0.72 to 0.78 because every claim in them is true and grounded, which is all a gate grades. Two rules follow. Match the sentinel at BOTH ends (and at the tail, by SENTENCE, so a post ending “…and the room went empty.” survives). And when a sentinel is the contract between a model and the code, enumerate the shapes it actually arrives in before trusting one: replayed over 470 shipped posts, the widened match caught exactly the 4 bad ones and nothing else.

A compose self-gate may only grade against a claim that passed a truth check upstream. [this repo: #1890, #2035, #2043] Every compose surface self-gates by scoring the take against the ground-truth blob built FROM the claim, so the gate answers “is this take faithful to the claim” and can never answer “is the claim true”. When the claim carries a wrong number the gate scores the post high and is CORRECT to (#1890: the post said “back at No. 1” off a chart reading #2, gate 0.91; #2035: the wire said 6 songs, the verify judge said 7, the post shipped saying 6 at 0.90). The invariant a new claim path must satisfy: before compose, the claim is either a FIRST-PARTY read (we fetched the chart/market/score ourselves) or it passed a verify verdict (and a CONTRADICTED verdict drops the claim, it does not soften it). The gate is never the backstop for claim truth — do not respond to a wrong-claim incident by tuning the gate. Audit note (2026-08-09): music_news satisfies it (live-chart settle

A DRY RUN must exercise the real SHIP PATH, not the model calls alone. [this repo: #2662/#2671] A compose surface has three stages: the compose, the deterministic guards, and the self-gate. A dry run that calls compose_* and *_score directly exercises two of them and SKIPS the guards, which live in the cog between them. That gap is not theoretical: the takeover board’s dry run passed 4 of 4 at 0.95, and every one of those takes would have been dropped by _unshippable before the judge ever saw it, because the board’s headline matched has_row_tally. The feature would have shipped, gone silent, and looked like a quiet chart. Only the code review caught it.

So a dry run of a gated surface calls the guard the cog calls, in the order the cog calls it, or it says plainly which stages it covered. The stronger form, where the surface allows it, is to drive the cog’s own compose method with a fake delivery. The rule generalizes past this repo: a “works end to end” claim is only as wide as the narrowest stage the run actually touched, and the stage most likely to be skipped is the cheap deterministic one nobody thinks of as behavior.

Tests

Tests pin behavior, and a regression test must fail first. [community; generalizes the golden rule] A test written for a fix runs RED on the pre-fix code or it pins nothing (#1886: asserts on shapes the gate rejected anyway pass whether or not the filter works). Assert outputs, state changes, emitted events — not call counts or internal ordering. A test needing three mocks to reach its assert usually means the pure core hasn’t been extracted yet (utils/ split).

The event loop

CPU work in an async path runs in asyncio.to_thread. [local incident; asyncio docs] The bot is one process with one event loop, and the Discord heartbeat runs on it. A Pillow render (about 1-2s each on Railway, a card can draw three), an image re-encode, or a pure loop over thousands of rows blocks the heartbeat. Past 10s the gateway drops the session, and the bot answers nothing while it reconnects. Measured 2026-09-28: 157 heartbeat blocked warnings in 30 days, almost all card renders and one O(rows x names) credit index. Call the sync builder as await asyncio.to_thread(build_number_card, ...), so a test that patches the name on the cog still works. A LayoutView with no timeout is safe to build in the thread. A cache-hit read counts too: an L1 hit returns without yielding, so a loop over many cached reads is ONE stretch however short each read is (the RIAA roster walk, #4291). The warning’s traceback names the blocking frame; see docs/OBSERVABILITY.md -> “Discord gateway health”.

Rules, checks, and corrections

Mechanize any rule that can be a check; a repeated correction demands it. [Cloudflare / HumanLayer / claude-md-templates] A convention worth keeping becomes a test, lint, hook, or CI gate — prose rules decay under context pressure, checks don’t (the test_event_schema ratchet, output_checks, the repo-root allowlist). When the owner corrects the same CLASS of mistake twice, that session ships the durable artifact — a check, else a rule here — in the same breath; “noted” without an artifact is how the third occurrence happens.

Review findings stay scoped to the patch. [OpenAI] Complexity is a real finding only when the machinery isn’t required by the task or a verified risk — don’t accept speculative abstractions, knobs, or parallel paths a reviewer imagines wanting. Unrelated cleanup a review surfaces gets its own issue, not bolted onto the PR.

Workflow hygiene

No user content or PII beyond the surface it serves. [Sentry] Real user messages or usernames-plus-content never land in commits, PR bodies, issues, or logs — the constitution’s data-minimization rule covers our own dev artifacts too. Repro fixtures use synthetic text.

Leave a trail a cold session can follow. [HumanLayer; this repo’s doc culture] Owner steers, incident lessons, and “why it’s built this way” go in the durable homes (the surface’s docs/ARCHITECTURE.md section, the tracking issue, the PR body) — not chat. The test: could a fresh session pick this up from the repo alone?

A scheduled check-in points at state; it does not CARRY state. [owner steer 2026-08; observed] list_triggers showed check-in prompts grown to 2,000+ words, one carrying an incident narrative, a pending owner decision, an unpushed local branch name, and three Axiom queries. That state lives nowhere else: not the issue, not the PR, not the repo. If the trigger fires into a dead session, or nobody reads it, the knowledge is gone. So the durable state goes on the tracking issue and the prompt says which issue to read and what to check — one or two lines. Same test as the rule above: the repo alone must be enough.

A PR that does not target main gets NO CI, and its check list still looks green. [measured 2026-08-10, #2266] .github/workflows/ci.yml triggers on pull_request: branches: [main]. Three cinema PRs were split into a hand-rolled stack, each based on the branch below it. claude-code-review.yml has no branch filter, so it ran on all three and reported success — and a check list holding one green claude-review and nothing else is, at a glance, indistinguishable from a fully green PR. None of the ci.yml checks had run on any of them. Nothing was red, so nothing announced the gap; the only reliable signal is the presence of the CI check runs by NAME, which is why this rule is “count the checks”, not “look for red”.

ci.yml posts ELEVEN checks, and a green subset is not a green suite. Count all of them: lint-typecheck, test (1) through test (8), coverage, and fts-postgres. lint-typecheck runs ruff, mypy AND the repo-linting tests (LINT_STYLE_TESTS in tests/_sharding.py), so a red one there is a real test failure, not only a lint slip. The coverage job is the one that enforces the 50% floor: each test shard measures only its own eighth of the suite, so coverage combines the eight and gates on the total. A run whose shards are green while coverage is missing has NOT been gated.

The shard COUNT lives in one place, .github/workflows/ci.yml (the matrix and the --shards argument must agree). If you change it, change this line and the CLAUDE.md Commands section too – a count that drifts turns “count the checks by NAME” into a rule nobody can apply.

There are two safe ways to split a large change, and hand-rolled base chaining is neither:

  1. Sequential PRs. Land one, git fetch origin main, branch the next off the updated main. Every PR targets main, so every PR gets the full gate. This is the default and needs no tooling.
  2. A real GitHub STACK, via the gh stack extension (public preview since 2026-07-30). GitHub then treats the chain as a stack rather than as PRs that happen to point at each other: “CI checks triggered by pull requests on your default branch run for all pull requests in the stack, not just the bottom one,” and “branch protection rules … are enforced on every pull request in the stack, even mid-stack pull requests that don’t directly target your default branch.” Stacks merge bottom-up; merging mid-stack merges everything below it and the PRs above “stay open and automatically re-target the stack’s base branch”. Commands: gh stack add <branch>, gh stack push / gh stack submit, gh stack sync, gh stack rebase, gh stack modify.

Recovering a stack you have already hand-rolled: retargeting the base to main is NOT enough on its own. A base change fires the edited event, which ci.yml does not listen to, so the PR keeps its empty check list. Retarget and then push — merging main into the branch fires synchronize, which CI does listen to. Expect the review NOT to re-run (the workflow skips synchronize); the earlier review still stands if the merge changed no code it read.

Sources: Stacked pull requests, About stacked pull requests, Stacked PRs CLI commands.

A WORKFLOW THAT COMMITS TO main RESTARTS THE LIVE BOT. Give it a watch-path exclusion. [#3340, 2026-09-15] Railway auto-deploys every push to main, and a redeploy is a full bot restart: a fresh Discord login, a re-sync of 21 slash commands per guild, and an SGO circuit breaker that trips open on boot. A committing workflow therefore costs one restart per run, and the cost is invisible – each deploy reports SUCCESS.

The Sleeper snapshot workflow ran hourly and committed sleeper-fantasy/data, which no module in the bot reads. That was about 20 restarts a day for data the running process never opens. On 2026-09-15 the churn earned a Cloudflare block on the Discord login call: the 14:26 deploy got 429/1015, the 14:49 deploy got a Cloudflare 500, both failed their healthcheck, and /menu answered “This interaction failed”.

Two rules follow:

Source: Railway watch paths.

A WORKFLOW THAT COMMITS DATA TO main ALSO GROWS EVERY CLONE. Keep scheduled data out of this repo. [#3945, 2026-10-08] The Sleeper job committed about 45 MB of changed JSON to main every hour. A cloud session clones the last 50 commits, and 33 of the last 50 were snapshot commits. Measured 2026-10-08: the clone was 1.9 GiB, and about 40 MB of it was code. The Sleeper bridge now lives in its own repository, mejasonmejason/sleeper-fantasy, and its README holds the details. The rule: generated data that changes on a schedule does not go on this repo’s main. The watchPatterns exclusion and the paths-ignore entries for sleeper-fantasy/data stay as a guard, and they cost nothing.

Shipping and closing out

These are the procedures behind CLAUDE.md’s always-loaded triggers. CLAUDE.md says WHEN; this section says HOW.

A NEW SCHEDULED SURFACE MUST BE GIVEN CALENDAR SLOTS, or it ships dark and silent. [owner decision 2026-08-31, #2315] Do this as part of shipping the surface, not after someone notices the silence.

schedule_calendar.calendar_hours treats a mood as calendar-managed the moment that mood’s key exists in the stored post_schedule JSON. For a managed mood it returns the surface’s assigned slots, and a surface with NO entry returns [] – which is deliberately distinct from None (no calendar at all, use the cog’s hardcoded pool). So in a guild that already has a managed calendar:

calendar_hours(...) -> []      # managed mood, no slot for this surface
due = []                       # ScheduledPoster._maybe_scheduled_post
return                         # every tick, forever

The cog’s own CHILL_TIMES / YAPS_TIMES are dead code there. Nothing raises, nothing is emitted, and no post is ever built, so the surface looks healthy at every other checkpoint: the cog loads, the channel is configured, the experiment resolves, the kill switch is off. sports_boards (#2293) sat like that from 2026-08-11, and the tell was zero rows in its own slot table, not an error anywhere.

So a new scheduled surface needs ONE of:

That is why the other recent surfaces were fine and this one was not: they are on the firehose list. Check which applies before you call the surface shipped, and put it in the PR’s test plan.

The fallback was proposed and REJECTED — do not re-propose it without new evidence. Making an absent surface fall back to its hardcoded pool would close the class, but it changes scheduling for every existing surface and could start posting something a mod believes is switched off. The owner’s call (2026-08-31) is that a silent-POST failure is worse than a silent-DARK one, and that the calendar stays the single source of truth for when a surface posts. The trap is closed by this written step, not by code.

The session-close miss sweep — the five questions. [owner steer 2026-08] Before handing a session back, run these in order and answer each in one line:

  1. The class sweep. Where else does this exact shape live? (The same question as “fix the class, not the instance” — this is the backstop for when it got skipped mid-task.)
  2. Scope I cut. What did I narrow, defer, or leave for later, and did I say so out loud?
  3. Coverage gaps. An untested branch, a fail-open path with no telemetry, a surface with no dashboard panel, a doc left stale.
  4. What surprised me. Anything the docs did not predict — usually the most valuable issue in the sweep.
  5. What I could not verify. A path the dry run did not reach, an assumption I shipped under.

Then file per the “Issues as a build log” bar: a real gap earns an issue, noise does not. Attach each to its epic with a p0–p3 label at creation. The body says what is missing, why it was out of scope, and what “done” looks like; cross-reference the PR that surfaced it. Report what you checked and deliberately did NOT file, not only the hits — a sweep that reports only hits reads as thorough while hiding what it dismissed. A clean sweep is a complete answer; say so plainly. Never file a follow-up as a substitute for finishing in-scope work; when unsure which you have, ask rather than file.

Worked example: the sweep on #2060 found the eval report is discarded, then found the ops monitor discards its report the same way — a class caught within minutes of the instance (#2061).

The post-ship verification Routine. [owner steer 2026-08: “sometimes we ship big changes that dont work or go silent”] A merge is not a landing. Railway auto-deploys main and the shipping session ends minutes later, so a change that fails to deploy or goes silent has nobody watching it.

BUILD IT AROUND A SESSION THAT HAS THE REPOSITORY (#3587). A Routine fired with create_new_session_on_fire: true gets sources: []. With no checkout there is no .claude/settings.json, so nothing grants it Bash or mcp__github__*, and it can neither read Railway and Axiom nor post its verdict. Seven of those reported nothing. The working shape, verified 2026-09-22 by a Routine that really posted:

  1. create_session with source_url set to this repository.
  2. create_trigger with persistent_session_id set to that session, plus run_once_at.
  3. A prompt that posts with mcp__github__add_issue_comment. An outbound curl WRITE is refused by an “External System Writes” classifier even from the repo-backed session; Bash curl is still right for the Railway and Axiom READS.

CLAUDE.md’s “Watch what you ship until it is VERIFIED” remains the binding rule: the shipping session still does the read by hand, and still hands off in writing what it cannot observe yet. The Routine is the second look, not the verdict.

When it applies: a new surface or cog, a new outbound integration, a change to what a LIVE surface posts or when, a schema/migration change, and any bug fix whose symptom was silence. Not docs, not a pure refactor, not a test-only change, not a tunable nudge.

How to build it: pick the SHAPE first — see “USE A ONE-SHOT. THERE IS NO CRON OPTION.” below, which is the binding rule and gives the two allowed shapes and their timing. Never a cron, whichever you pick.

Whatever the shape, the prompt is a COMPLETE standalone instruction naming the PR, the issue, the surface, the exact check, and the pass condition — and it points at the issue for state (see the rule above). A Routine bound to a repo-backed helper session starts from nothing, so it needs that completeness most. Allow time for Railway to build AND for the surface to reach a slot — about two hours after merge, and the Routine gets a single attempt.

The fired session inherits Railway and Axiom access — do not override the environment. The verification steps below need both: Railway answers “did it deploy”, Axiom answers “is it alive”. The environment carries RAILWAY_API_TOKEN, RAILWAY_SERVICE_ID and AXIOM_API_KEY, and create_trigger gives the fired session the calling session’s environment when you leave environment_id unset. So leave it unset. Passing a different environment_id moves the Routine to an environment that may not hold these keys, and the verification then fails for a reason that has nothing to do with the change it is checking.

Verified 2026-08-09 from a session in this environment: the Railway GraphQL service query returned HTTP 200, and the Axiom _apl query returned HTTP 200 with 201,337 rows over 24h. Read AXIOM_API_KEY straight from the environment; it does not need a Railway round-trip.

The fired session is HEADLESS, so it must never meet a permission prompt. A fresh-session Routine starts with no person at the keyboard. A tool call that asks for approval gets no answer, so the session stops at that call and the check never completes. The Routine does not report a failure — it goes quiet, which is the same symptom the Routine exists to catch.

.claude/settings.json sets permissions.defaultMode to bypassPermissions, so Claude Code runs every tool with no prompt. Do not replace this with a list of allowed tools. A list only covers the tools we predicted; the first tool outside it stops the session, and it stops silently. That is what happened before 2026-08-09.

THAT SETTING DOES NOT REACH A SESSION THE ROUTINE CREATES, AND THIS IS WHY THE ROUTINES STOPPED REPORTING (#3587). .claude/settings.json lives in the REPOSITORY, so it applies only to a session that has the repository. A session CREATED by a firing — create_new_session_on_fire: true — does not have one: its session_request.config.sources is [], and create_trigger exposes no parameter to set it. With no checkout there is no .claude/settings.json to read, so that session falls back to permission_mode: auto with an EMPTY auto_mode_allow — exactly the “list of allowed tools” failure this paragraph warns about, arriving by a route the paragraph did not anticipate.

This is about the session, not about Routines. A Routine bound with persistent_session_id does not create a session; it DELIVERS into one that already exists, and that session keeps its own sources. So a Routine bound to a session made by create_session with source_url has the checkout and the allow list. That is the working recipe, and the paragraphs below build it.

Measured 2026-09-22. A probe Routine whose entire prompt was “write this file, post it with this curl, report the code, stop” ran 21 seconds, finished IDLE, and posted nothing. Two verify sessions sat REQUIRES_ACTION / BLOCKED for hours with the command in pending_action and used_tokens: 0: #3588’s on curl ... raw.githubusercontent.com/.../CLAUDE.md, #3564’s on curl ... backboard.railway.com/graphql/v2. The first is the tell — a session WITH the repo would Read CLAUDE.md off disk; that one had to fetch it over HTTP because it had no checkout, and the fetch is what blocked it.

THE REPOSITORY IS THE VARIABLE, NOT THE PERMISSION MODE. This section used to say the fix was an account-level permission grant that no session could make. That was wrong. Measured 2026-09-22: a session created with create_session(source_url=<this repo>) runs the SAME permission_mode: auto, and from it a curl GET to api.github.com/user and a curl POST to the Axiom _apl endpoint both returned 200. A Routine bound to that session with persistent_session_id then fired and posted a real comment on #3587. The difference is the checkout, which brings .claude/settings.json with it.

One rule survives the fix: an outbound curl WRITE is still refused, by an “External System Writes” classifier that a repo checkout does not turn off. The probe’s curl POST to the GitHub comments endpoint was denied with Reason: [External System Writes], and the same session then posted the comment through mcp__github__add_issue_comment. So the prompt posts its verdict with the MCP tool, never with curl -X POST.

AND THE HELPER MUST STILL BE ACTIVE WHEN THE ROUTINE FIRES. An archived session is gone as far as a trigger is concerned. Measured 2026-09-22 in two steps: binding a trigger to an already-archived session is REFUSED at creation (“only active sessions can be bound to triggers”), and a trigger bound while the session was active, whose session then archived, FIRED AND DIED — run FAILED after 8 milliseconds with ended_reason: auto_disabled_session_gone, no session woken, nothing posted. A session archives on its own once idle, and the working probe posted only because its session was still RUNNING minutes after creation. So a Routine placed two hours after a merge has no evidence behind it and one measurement against it. unarchive_session does not rescue it: a one-shot that fired and failed does not fire again.

Note the boundary: the classifier reads the DIRECTION of the data, not the HTTP verb. An Axiom _apl query is a POST and it was allowed, because it reads. Not yet measured: the Railway GraphQL read from a Routine-fired turn. The Axiom half of the liveness check is proven; the Railway half is not. If a verify run comes back blocked on backboard.railway.com, that is the untested leg, and it should be recorded here.

Do the verification by hand in the shipping session before it ends all the same, per CLAUDE.md’s “Watch what you ship until it is VERIFIED”. The Routine is the second look.

Two traps, both seen in this repo:

The named permissions.allow list stays in the file as a fallback, for the case where a policy turns bypass off. The guards that keep a headless session safe are not permission checks, and they still hold: no direct push to main, the owner merges protected-path changes, /debug/query is read-only, and the production database is unreachable from a session.

What to check, in order:

  1. Did it deploy? The Railway deployment for the merge commit is SUCCESS, not a build failure or crash loop. SKIPPED is not a deploy (#3112): Railway waits for every check suite on main, so a failed GitHub Pages job can skip the bot’s deploy. Read the deployment’s status and skippedReason. If it is SKIPPED, redeploy that commit.
  2. Is it alive? The surface’s events appear AFTER the deploy time — compared against the rate BEFORE the change, not against zero. A surface emitting nothing looks identical to a quiet hour, which is how a silent ship survives a naive check. ['tootsies'] | where surface == '<surface>' | summarize count() by bin(_time, 1h)
  3. Is it healthy? No new error rows from that source, ok rates steady, and the metric the change was supposed to move actually moved in the predicted direction.

It must reach a terminal state. A Routine that keeps firing trains everyone to ignore it. Verified → tell the owner what the numbers show. Broken → fix it or open the revert, then tell the owner. Inconclusive by ~48h after the merge → hand it over in plain words: what you checked, what you could not confirm, what you would look at next.

Every one of those is a REPORT, not a deletion. Under the one-shot rule below the Routine has already expired on its own, so nothing needs deleting — and a fired session normally could not delete anything anyway. delete_trigger belongs to the shipping session and to the close-out sweep, never to the Routine’s own prompt.

USE A ONE-SHOT. THERE IS NO CRON OPTION. [measured 2026-08-29] A Routine must expire without anybody calling delete_trigger, because the session that would call it usually cannot. create_trigger stores the calling session’s MCP connector grants, and a Claude-on-web session normally holds none to pass through (the tool says so in a warning on the response). A session CREATED by the firing then has NO mcp__* tools at all — list_triggers and delete_trigger do not exist in it — so a cron Routine fires forever no matter what its prompt says. [verified 2026-08-10, PR #2275] A repo-backed session bound with persistent_session_id does hold them, but use a one-shot there too: a cron Routine still outlives its purpose, and nothing in the prompt makes it stop.

The evidence that this is the NORMAL case, not the edge case: a sweep on 2026-08-29 found 11 stale verify Routines, every one of them cron, every one carrying a self-delete step, and not one of the fired sessions holding a single mcp__* tool. The oldest was 4 days past its own 48h deadline (#2547); others covered #2620, #2636, #2644, #2645, #2650, #2658, #2660, #2667, #2671 and #2680. They fired on schedule, correctly judged themselves “not terminal yet”, and re-armed — forever. Nothing was broken. The recipe was.

A FRESH-FIRED SESSION HAS NO GITHUB TOOLS; A REPO-BACKED ONE DOES. [reproduced 2026-08-29 shipping #2691; corrected and re-measured 2026-09-22] A create_new_session_on_fire session’s allowed_tools carries Task, Bash, Glob, Grep, Read, … and not one mcp__ entry, so a prompt reaching for mcp__github__* calls a tool that does not exist and the run stops there, silently. That is a property of the MISSING CHECKOUT, not of Routines: a session created by create_session with source_url loads .claude/settings.json, whose allow list carries mcp__github, and it posts through mcp__github__add_issue_comment normally.

Do NOT write the prompt for curl or gh as a substitute. An outbound curl WRITE is refused by an “External System Writes” classifier, measured in the same session that then posted successfully through the MCP tool, so the gh-over-Bash advice this paragraph used to give could not have worked either. Bash curl is correct for READS (the Railway GraphQL and Axiom _apl endpoints both return 200 from a repo-backed session); the MCP tool is correct for the post.

Cron is not offered even behind a “only if you confirmed the connectors” condition. That condition is exactly what the old recipe had, and it is what failed: an exception a creating session has to notice in a tool response is an exception nobody reads. The retry cron buys is small — a Railway build takes minutes — and it is not worth a watcher that cannot stop. Pick one of two shapes:

  1. send_later, bound to the SHIPPING session — use when you will keep that session open across the window (you are still working in it). It is a one-shot, so it disables itself after firing, and it fires into a LIVE session that does hold the MCP tools, so that session can re-check, re-arm another send_later while the answer is genuinely pending, and stop when it is not. That chain is the retry mechanism, and it terminates because a human-facing session decides when to stop. #2684 used it; it closed on the second firing. Measured over ~9h and ~10 firings: a self-bound send_later keeps reaching the same session while that session is ACTIVE, across container restarts — the tool documents that delivery survives them. It is a CLOSED session that breaks it: if the session is gone when the delay expires, the check is simply lost, and nothing reports the loss. So when you are not sure the session will still be there, use shape 2.

  2. run_once_at, bound to a REPO-BACKED helper session — the default when the shipping session will end, and the safe answer whenever shape 1 is in doubt. Build it in two steps: create_session with source_url set to this repository, then create_trigger with persistent_session_id set to that session’s id and run_once_at set to the check time. The helper session holds the checkout, so it holds Bash and mcp__github__*, which is what the verdict needs. It gets ONE attempt and terminates by construction. The prompt must say that the Routine is a one-shot, that it must not try to delete itself, and that it posts its verdict with mcp__github__add_issue_comment and not with curl.

    Timing is this shape’s unsolved problem. A verification wants to run about two hours after the merge, so a slow Railway build and a late cadence slot cannot burn the single attempt. But the helper session archives once it goes idle, and a Routine firing into an archived session dies in 8 milliseconds with auto_disabled_session_gone (measured 2026-09-22, see above). The only firing that has ever worked landed minutes after the helper was created, while it was still RUNNING. So this shape is best-effort: use it, and never let it be the verdict. The written hand-off on the issue is what carries the obligation.

    Never use create_new_session_on_fire: true for this. That shape fires with sources: [], so it has no checkout, no .claude/settings.json, no mcp__* tools and no way to post. Seven of them reported nothing (#3587).

A one-shot that cannot conclude must HAND OFF, not just expire. One attempt means the surface may still have been quiet — which is the state this whole section exists to tell apart from a dead surface, so an expired trigger is NOT by itself a terminal state. The prompt must require the fired session to end by reporting to the owner in plain words: what it checked, what it could not confirm, and what it would look at next. Turn on the Routine’s completion notification so that report actually arrives. Deploy status and error counts are conclusive on a single attempt; only the “did the surface fire” half can come back unresolved.

And NOTHING catches that half for you, so an unresolved liveness check MUST become an issue. surface_dark is not the net. It reads JUDGE SCORES, gated on qs.n >= DARK_SCORE_MIN and below_rate >= DARK_SCORE_RATE — at least five SCORED posts, 90% of them under the ship floor. A surface producing NOTHING has no scores, clears no minimum, and trips nothing. scripts/ops_monitor.py says so about itself twice: “a desk refusing everything would show no scores at all and read as a quiet news day”, and “lane_dark and surface_dark both read judge scores, so neither can see it, and the lane just goes quiet like a slow chart week”. Zero events is exactly the shape both sentences describe. [PR review catch, 2026-08-29: an earlier draft of this paragraph claimed the monitor covered it, which would have told a future session it was safe to let the check expire — a false reassurance on the one failure mode this section exists for.] So when the single attempt could not confirm the surface fired, FILE THE FOLLOW-UP. Do not leave a watcher running, and do not assume anything downstream is watching. #2694 is the worked example.

Name it for one thing anyway. verify-pr-<number>, with the merge time as literal text in the prompt. Neither shape needs the name to delete itself, but the sweep below needs to recognise it, and so does anyone reading list_triggers later.

Keep the sweep as the backstop. Self-deletion cannot be guaranteed, because the agent that deletes the Routine is the same agent that might be broken. Run list_triggers during the session-close sweep and delete stale ones. If you find one you cannot account for, say so rather than deleting it silently.

Before deleting a verify Routine, CHECK ITS SUBJECT rather than clearing the list: confirm the PR merged and the surface is healthy (no unrecoverable errors, no surface_dark, the surface still posting). Deleting a watch on something actually broken is worse than leaving it. And honour any unresolved item the Routine’s own prompt names — the #2667 one could never observe its new album source inside a 9-day dedup window, and its prompt said to file a follow-up if it closed unconfirmed, so the cleanup filed #2694 instead of dropping the question.

A stale Routine is easy to spot in the list_triggers payload: it has a cron_expression, and its session_context.allowed_tools contains no mcp__* entry — that combination cannot terminate.

Dashboards and monitors are part of instrumenting a surface. [owner steer 2026-08] An event nobody graphs is a log line, not a metric. scripts/axiom_setup.py is the single source of truth for both — dashboards upsert by uid, monitors by name, so it is idempotent (python -m scripts.axiom_setup --dashboards --monitors). Every new event gets a PANEL (volume, ok rate, and the latency percentile when it carries duration_ms). Then ask whether it earns a MONITOR: a threshold worth waking someone for — a hard blackout, an ok rate through the floor, a quota about to run out — rather than a number worth looking at. Say which you added and why you skipped the other; “no monitor, the ops-monitor finding covers it” is fine, silence is not.

This applies to a CHANGED surface too. When a change moves what a surface emits, the panel and any monitor threshold reading the old shape are now wrong, and a stale panel reads as healthy while it measures nothing. Re-point them in the same PR and state which metric moves and in which direction. Worked example: #2052 drove has_link to False across ~23% of crossposts — an expected step down that would otherwise look like a compose fault.