tootsies
a discord bot for the tootsies server. ask, recap, discuss, ship features by typing.
Project maintained by mejasonmejason
Hosted on GitHub Pages — Theme by mattgraham
Billboard source research
Source ledger for the Billboard frontier — alerts (big jumps / debuts / changes),
forecasts (first-week + Hot 100 projections), and reports (chart weeks, resolved
markets). Every claim below was probed live from a session on 2026-08-03; the
probe and its result are recorded inline so a future session can re-run it rather
than re-derive it. Numbers are what the probe actually returned, not estimates.
This is research, not a design. It says what exists, what it costs, what it can’t
do, and the two decisions that are the owner’s to make. It deliberately does not
pick an architecture.
1. What we have today, and the gap
The repo already has a lot of chart-adjacent machinery. None of it reads Billboard.
| Module |
What it actually is |
Billboard? |
utils/kworb.py |
Spotify US/global daily+weekly, Apple songs/albums, iTunes, radio airplay |
No — proxies |
utils/luminate.py |
Luminate consumption units, read off Kalshi’s public API |
No — the input Billboard filters |
utils/chart_data.py |
Wikipedia discography tables — historical peaks |
No — history, not this week |
utils/wiki_hits.py |
Billboard Year-End charts off Wikipedia |
Annual only |
utils/music_news.py + cogs/music_news.py |
The newsroom: trusted wire accounts → verified numbers posts |
Relays Billboard claims |
So today a Billboard fact reaches Toots exactly one way: an X account says it and
the newsroom tries to verify. There is no first-party Billboard chart read
anywhere in the codebase.
The gap has a measurable cost already
The newsroom’s chart-claim gate (cogs/music_news.py:_live_chart_position,
event music_news_chart_lookup) verifies a wire claim against live chart rows.
Those rows come from kworb — Spotify and Apple. So a Billboard claim off
@billboardcharts gets checked against a different chart.
Axiom, last 30 days:
['tootsies'] | where event == 'music_news_chart_lookup'
| extend r=tostring(parse_json(detail)['reason'])
| summarize n=count() by ok, r | sort by n desc
ok | reason | n
True | | 88
False | absent | 16
False | ambiguous | 9
False | unresolvable | 5
118 lookups, 30 unresolved (25%). absent is the one that matters: the claim
named a song the Spotify/Apple chart didn’t have — which for a Billboard claim is
often not a false claim, just the wrong chart. A real Hot 100 feed makes this gate
check the chart the claim is actually about.
Volume context (7d): music_news_scored 813, music_news_posted 77,
chart_fetch 927 with 927/927 ok — kworb is healthy; it’s just answering a
different question.
2. The source ledger
Verdict: the primary read. Cheap, complete, and much better than its reputation.
The charts render server-side in the HTML. No JS, no API key, no rendering.
curl -s -A "Mozilla/5.0" https://www.billboard.com/charts/hot-100/
# http=200 bytes=3,085,728 time=0.44s
Rows split on <ul class="o-chart-results-list-row — exactly 100, each carrying
rank (also in data-detail-target), title, artist, LW, PEAK, WEEKS:
['1', "Choosin' Texas", 'Ella Langley', 'LW','1', 'PEAK','1', 'WEEKS','40']
['2', 'I Knew It, I Knew You', 'Taylor Swift', 'LW','2', 'PEAK','1', 'WEEKS','7']
['50','misery.', 'pupsies', 'LW','62', 'PEAK','50', 'WEEKS','3']
That field set — rank, last_week, peak, weeks_on — is everything the alert
lane needs. Jumps, debuts (LW empty), re-entries, new peaks, weeks-at-#1
streaks, and falls are all derivable from one fetch, no state required.
One parser covers the whole family. Probed, all 200 with Week of August 1, 2026:
| Slug |
Rows |
|
Slug |
Rows |
hot-100 |
100 |
|
billboard-200 |
200 |
billboard-global-200 |
200 |
|
billboard-global-excl-us |
200 |
artist-100 |
100 |
|
streaming-songs |
50 |
radio-songs |
50 |
|
top-album-sales |
50 |
digital-song-sales |
25 |
|
country-airplay |
60 |
rap-song |
25 |
|
|
|
404s (slug is different, not missing): global-200, pop-airplay,
tiktok-billboard-top-50, songs-of-the-summer.
History is free. /charts/hot-100/2026-07-25/ → 100 rows, Week of July 25,
2026. Any past week is addressable, so week-over-week diffs, backfill, and “when
did this peak” need no stored history — though storing is obviously cheaper than
refetching 3MB.
Costs and risks. 3MB/fetch is the real price (~30MB/week for the full family
at one fetch per chart per week — trivial). Markup is WordPress/PMC class-name
driven, so a redesign breaks the parse: it needs the utils/box_office.py /
utils/kworb.py guard treatment (breaker + limiter + TTL cache + fail-open) and
an OkRate finding so a silent shape change surfaces. Fail-open, never fabricate
— a dropped chart beats an invented rank.
⚠️ Governance — this is an owner call, see §5.
2.2 HITS Daily Double — the forecast layer, via an open JSON API
Verdict: the highest-value find. This is the forecast source, and it does not
need scraping at all.
hitsdailydouble.com is a Next.js SPA (336KB shell, one text node — unscrapable
by fetch, and Chromium is blocked by this sandbox’s proxy). But its JS bundle
leaks the CMS:
grep -rhoE 'cdn\.sanity\.io/[a-z]+/[a-z0-9]+/[a-z0-9_-]+' hits.html
# cdn.sanity.io/images/8aky18h3/production
HITS runs on Sanity, project 8aky18h3, dataset production — and the GROQ
query API is public and unauthenticated:
curl -G https://8aky18h3.apicdn.sanity.io/v2023-05-03/data/query/production \
--data-urlencode 'query=*[_type=="chart"]|order(_updatedAt desc)[0]'
# HTTP=200 118,610 bytes "ms": 35
35ms JSON. No key, no quota, no scrape, no rendering.
Relevant document types (from array::unique(*[]._type)):
| Type / chart_type slug |
What it is |
Cadence observed |
chart → hits-top-50 |
Building chart: LW, TW, artist | album, label, total units, %chg, album sales, song sales, streaming units — plus a MARKETSHARE: UMG 56% SME 28% WMG 16% header |
Jul 24, 25, 31 |
chart → midweek-20 |
Midweek Top 20 projection: LW, TW, artist | album, label, projected units, projected album sales |
Jul 14, 24, 28 (~Tue) |
building_chart |
Mediabase radio building chart: plays this/last week, %chg, rank moves |
_updatedAt 3h before probe |
add_chart |
Radio adds this week |
_updatedAt 4 min before probe |
chart → ytd-albums, song-streams, song-revenue, ytd-country-albums, ytd-latin-song-streams, ytd-song-streams, 2024/2025-year-end-albums |
YTD + year-end |
periodic |
Sample — midweek-20, the projection PopBase quotes as “according to HITSDD”:
3 1 MORGAN WALLEN | I'M THE PROBLEM BIG LOUD/MERCURY (UMG) 81,000 1,900
2 2 OLIVIA RODRIGO | YOU SEEM PRETTY SAD… GEFFEN (UMG) 79,500 16,000
NEW 4 CHARLI XCX | MUSIC, FASHION, FILM ATLANTIC (WMG) 75,000 50,000
Two shapes per doc: chart_data (a TSV string) and chart_entries — a
structured array of {artist, album, columns:[{label, value}]}. Prefer
chart_entries; the labels are self-describing, which is exactly what
docs/INTEGRATIONS.md asks for, and it survives column reordering.
This closes the forecast ask directly: first-week album projections, radio
building/adds (the leading indicator for Hot 100 airplay movement), and label
marketshare — all as typed JSON.
⚠️ Governance — see §5.
2.3 GitHub JSON mirror — the free fallback
mhollingshead/billboard-hot-100 publishes every Hot 100 chart in history as JSON.
curl https://raw.githubusercontent.com/mhollingshead/billboard-hot-100/main/recent.json
# http=200 11,841 bytes 0.63s
{"date":"2026-08-01","data":[{"song":"Choosin' Texas","artist":"Ella Langley",
"this_week":1,"last_week":1,"peak_position":1,"weeks_on_chart":40}, …]}
Same field set as the scrape, 11KB instead of 3MB, and raw.githubusercontent.com
is already reachable and already a dependency (GITHUB_TOKEN, utils/github.py).
CORRECTION (re-probed 2026-08-04): it does NOT lag a week. The first pass here
said “it lagged a week at probe time” — that was wrong. On 2026-08-04 the mirror
served 2026-08-01 and so did billboard.com; neither had Aug 8. What the mirror
trailed on 2026-08-03 was the wire announcement, which billboard.com trailed
identically (§3).
How it actually works (read from the repo, not assumed): it is a crawler on
cron: '0 4 * * *' — it reads billboard.com once a day at 04:00 UTC, and GitHub’s
scheduler runs it late (the badge showed 07:26). One commit per day, `Crawled on
`. 3,548 charts back to 1958-08-04.
**Three real limits.** (1) **Hot 100 only** — no Billboard 200, no Global 200, which is
most of the ask. (2) **On flip day it lags** — Billboard publishes during the US
afternoon and the crawl doesn't run until the next 04:00, so it can be ~17h behind
exactly when a new chart matters. (3) It's one volunteer's cron; when it breaks, it
breaks silently and there's no one to page.
**Verdict: NOT USED (owner call, 2026-08-04).** The full arc, recorded so no session
re-derives it: wired as a fallback → promoted to the primary (to reduce billboard.com
reads) → demoted back → **removed entirely**. The mirror is itself a once-daily crawler
of billboard.com, so at every step it added a hop, a volunteer single point of failure,
and flip-day staleness — and never added data. When Billboard can't be read the bot
yields nothing, loudly (`source_blocked`), rather than serving second-hand rows.
Still noted as a cheap manual *backfill* (every chart since 1958-08-04 as JSON) if a
historical analysis ever wants it.
### 2.4 Kalshi — correcting a premise
**The brief assumed Kalshi as a forecast source for Billboard. For the weekly
charts, that is not currently true, and this is worth knowing before anything is
built on it.**
Billboard-settled series exist — `TOPSONG` (Hot 100 #1), `TOPALBUM` (Billboard 200
#1), `KXRANKLISTBB100ARTIST`, `KXTOP10BILLBOARD`, `KXBILLBOARDGENRE`, and a
`BILLBOARDPEAK-*` family of per-song peak ladders. But they are **dormant**:
```
TOPSONG events=0
TOPALBUM events=0
KXTOP10BILLBOARD events=1 (annual: "top 10 hit this year?")
KXRANKLISTBB100ARTIST events=3 most recent: KXRANKLISTBB100ARTIST-26MAR07
BILLBOARDPEAKUS events=0
KXBILLBOARDGENRE events=0
```
The weekly "who's #1 on the Hot 100" market last listed for the week of **March 7,
2026** — five months before this probe — and only ever ran three weeks.
**What Kalshi *is* good for is already integrated:** the Luminate album-units
ladders behind `utils/luminate.py` + `utils/music_markets.py`. That's live and
load-bearing. It is a **units** forecast, not a **chart-position** forecast.
**Verdict: do not build a Billboard forecast lane on Kalshi.** Worth a cheap
periodic check for the weekly series waking up (it would be a good market and
`music_markets.py` currently filters Billboard-settled markets *out* by design —
`_is_luminate_family`). Until then, HITS is the forecast source and Kalshi is the
units source.
### 2.5 kworb — already integrated, and it is not a Billboard mirror
Worth stating because it's easy to assume otherwise: `kworb.net/charts/` is
iTunes / Spotify / Apple / Shazam / Deezer / YouTube / radio. `kworb.net/charts/billboard/`
is **404**. kworb carries no Billboard mirror. It stays what it is — the daily
streaming proxy — and it's healthy (927/927 ok, 7d).
### 2.6 X accounts — already well covered; grow by evidence, not by guessing
The repo's curated lists are genuinely good and already cover the chart-stats
authorities. `utils/music_news.py:_TRUSTED` carries, with `relay_number=True`:
`chartdata`, `billboardcharts`, `billboard`, `chartmastersorg`, `touringdata`,
`hitsdd`, `talkofthecharts`, `headlineplanet`, `usradioupdater`, `hiphopnumbers`,
`kalshi_music`, `dailyrapfacts`, `top100kpop`. `utils/pop_stories.py:MUSIC_WIRE_ACCOUNTS`
adds the culture wires (`popbase`, `popcrave`, …).
**I could not probe handle liveness from this session** — `api.twitterapi.io`
returns `status: error, "user not found"` for even `popbase` through the sandbox
proxy, so any liveness claim from here would be an artifact of the sandbox, not a
fact about the account. Left unverified rather than guessed.
The right instrument already exists and is better than a one-off probe: the
newsroom emits `music_news` for a numbers story from an **untracked** account, so
production self-reports its own gaps. The honest recommendation is to **read that
signal** rather than bulk-add handles:
```
['tootsies'] | where event == 'music_news' | summarize n=count() by handle | sort by n desc
```
(Note: that query returned empty over 30d at probe time — `music_news` isn't
firing, while `music_news_scored` shows 813 in 7d. Either the untracked-account
branch isn't reached or the event isn't carrying `handle`. **Worth a look — the
gap-reporting mechanism the docs describe may not actually be reporting.**)
The structural point: **X is a discovery and timing signal, not a data source.**
It tells you a chart moved before any feed does (§3). The number should still come
from §2.1–2.3. That's already the newsroom's design; a Billboard feed just gives
its verification gate the right chart to check against.
---
## 3. Cadence — the finding that shapes the alert lane
Timing is the whole design constraint for alerts, and the sources disagree.
Probe, **Monday 2026-08-03 22:00 UTC**:
| Source | Chart it was serving |
|---|---|
| `@PopBase` on X | **Aug 8, 2026** — posted ~5h earlier (~17:00 UTC) |
| `billboard.com/charts/hot-100/` | Aug 1, 2026 |
| GitHub JSON mirror | Aug 1, 2026 |
**The wire beat both feeds by roughly a day.** Billboard's own chart pages hadn't
flipped yet; the mirror follows the pages, so it was behind too.
The consequence: **an alert lane that only polls a feed is structurally last.**
By the time billboard.com flips, the moment has been on the timeline for a day and
the newsroom has likely already posted on it. The two signals do different jobs:
- **X (`@chartdata`, `@billboardcharts`, `@PopBase`)** — *when* something happened.
Fast, partial (usually top 10 only), and needs verification.
- **billboard.com / mirror** — *what exactly* happened, all 100–200 rows, exact
LW/peak/weeks. Authoritative, ~a day late.
So the honest framing for the alerts ask: the wire is the trigger, the feed is the
truth, and the interesting window is the gap between them. Reconciling the two —
"the wire said this, here's the full row when it landed" — is a real surface that
neither source supports alone. That's a design decision, not a research finding,
so it stops here.
Cadence summary: Hot 100 / Billboard 200 refresh **weekly** (chart date Saturday;
pages observed flipping Tue-ish). HITS `midweek-20` lands ~**Tuesday**;
`hits-top-50` ~Thu/Fri; HITS radio `add_chart` / `building_chart` update
**multiple times daily** (4 minutes and 3 hours stale at probe). Polling
billboard.com more than a few times a day is waste; polling HITS radio hourly is not.
---
## 4. "Scraping is slow — can we do better?"
Yes, substantially. Only one of the four sources needs scraping at all.
| Source | Mechanism | Latency | Payload | Key |
|---|---|---|---|---|
| HITS (Sanity GROQ) | **JSON API** | **35 ms** | 119 KB | none |
| GitHub mirror | **static JSON** | 0.63 s | 12 KB | none |
| Kalshi | **JSON API** | — | — | none *(already integrated)* |
| billboard.com | HTML parse | 0.44 s | 3 MB | none |
The forecast layer — the part that felt like it would need a headless browser —
is a **35ms JSON call**. The only HTML parse left is billboard.com, and at 0.44s
for a weekly-refreshing resource it is not a latency problem; cached behind
`durable_cache` at the chart's real cadence it is nearly free. What it *is* is a
**fragility** problem (class-name-driven markup), which is a guards-and-telemetry
answer, not a speed answer.
**Rendering (Playwright/Chromium) is not needed for anything here.** Worth
recording, since it looked like it would be.
---
## 5. Two decisions that are the owner's
Both are the reversibility call from `CLAUDE.md` — external-facing, reputational,
not cleanly undoable once we're on someone's logs. Flagging, not deciding.
**a) billboard.com's robots.txt names 25 AI agents. — OPEN, and deliberately left as a
posture question rather than an architecture one.**
The full list, re-read 2026-08-04, is broader than the first pass reported. It blocks
every Claude agent (`anthropic-ai`, `ClaudeBot`, `Claude-Web`), the OpenAI ones
including the **user-directed** `ChatGPT-User` and `OAI-SearchBot`, plus `GPTBot`,
`PerplexityBot`, `CCBot`, `Google-Extended`, `Meta-ExternalAgent`,
`Meta-ExternalFetcher`, `Bytespider`, `DeepSeek`, `Applebot-Extended`, `Diffbot`,
`YouBot`, `Timpibot`, `cohere-ai`, `omgili`. `User-agent: *` allows `/charts/` — only
wp-admin, search and previews are disallowed.
**Where that leaves us.** We are not any listed agent: we send `TootsiesBot/1.0` with a
contact URL, and the `*` rules permit the path. Every probe returned 200; we have never
been blocked. A list that enumerates the user-directed fetchers is not ambiguous about
intent, so this is worth being conscious of — but it is a question about how we behave,
not a reason to contort the data layer.
**What we do about it, concretely:** an honest contactable User-Agent, a hard cache on a
weekly resource (~1 read per chart per week), a tiny rate limit, and loud block detection
(§2.1) so that *if* a 403 ever comes it is unmissable. There is deliberately no
stand-in source (§2.3). That is the whole posture.
**What we deliberately do NOT do:** spoof a browser User-Agent. That is the one option
that would be genuine evasion, and nothing requires it.
**b) HITS' robots.txt disallows `/api/` and `/studio/`. — RESOLVED (owner call,
2026-08-03: proceed).** Kept here because the reasoning is the precedent for the next
source of this shape.
The concern was over-stated on the first pass, and the checks that actually mattered were
run before `utils/hits.py` shipped rather than assumed:
- **The directive doesn't cover the host we read.** `Disallow: /api/` is on
`hitsdailydouble.com`; the data is on `apicdn.sanity.io` — a different host with its
own robots. And reading one known endpoint a few times a day isn't crawling; robots.txt
governs automated traversal.
- **Nothing is being bypassed.** No auth, no token, no rate gate. The project id
`8aky18h3` is published in their own client-side bundle and their images are served
from `cdn.sanity.io/images/8aky18h3/production` — their own frontend makes the
identical request from every visitor's browser. We read `apicdn` (the cached edge their
site uses), not the uncached origin.
- **The paywall is a magazine, not the data.** The Stripe links are labeled "Magazine
Subscription" / "Subscribe to Mag", `/sales` serves 200 anonymously, and **no document
in the dataset carries a gating flag — 0 of 234 chart docs.** An earlier `gatedAt`
sighting was a false positive: it matched Next.js's `navigatedAt` router internals, not
a paywall field.
**The obligation that comes with it is attribution.** Every HITS-sourced number is
credited to HITS by name (`hits.PROJECTION_ATTRIBUTION`), baked into the rendered block
rather than left to a prompt that a caller might not pass. The newsroom already names its
DATA sources (Luminate, Billboard) while never naming a reposting account; HITS is a data
source, so it gets named. Citing them is what makes this legitimate rather than
laundering their work.
Reasonable middle ground for both: treat these as **read-sparingly** sources —
fetch at their real cadence, cache aggressively, attribute the data source in the
voice (the newsroom already attributes Luminate/Billboard and never the reposting
account), and keep the read volume low enough that we're a rounding error.
---
## 6. What the research supports, per the three asks
Stated as findings, not a plan.
**Alerts (jumps / debuts / changes)** — fully supported by §2.1 with no new
infrastructure: `rank/LW/peak/weeks` on every row makes jumps, debuts,
re-entries, new peaks and streaks pure derivations, and dated URLs make
week-over-week diffs free. The open question is not data, it's **timing** (§3):
the wire is ~a day ahead of every feed.
**Forecasts** — §2.2 is the answer, and it's better than expected: HITS midweek
projections + building chart + radio adds, as 35ms typed JSON. **Not Kalshi** for
chart position (§2.4, dormant since March); Kalshi stays the *units* source it
already is.
**Reports** — §2.1 + §2.3 together: any chart, any week back to 1958, plus the
existing Luminate settlement path for resolved markets. This is the best-covered
of the three and needs the least new work.
**Biggest single unlock:** the newsroom's chart-verification gate stops checking
Billboard claims against Spotify charts (§1). That's 30 unresolved lookups in 30
days, and it's the cheapest real win on this list.
---
## Re-running the probes
```bash
# Billboard: any chart, any week
curl -s -A "Mozilla/5.0" https://www.billboard.com/charts/hot-100/2026-07-25/ \
| grep -c '<ul class="o-chart-results-list-row' # -> 100
# HITS: chart types, then the newest projection
curl -sG https://8aky18h3.apicdn.sanity.io/v2023-05-03/data/query/production \
--data-urlencode 'query=*[_type=="chart_type"].slug.current'
curl -sG https://8aky18h3.apicdn.sanity.io/v2023-05-03/data/query/production \
--data-urlencode 'query=*[_type=="chart"]|order(_updatedAt desc)[0]{_updatedAt,chart_entries}'
# GitHub mirror
curl -s https://raw.githubusercontent.com/mhollingshead/billboard-hot-100/main/recent.json
# Kalshi: is the weekly Hot 100 market awake yet?
curl -s "https://api.elections.kalshi.com/trade-api/v2/events?series_ticker=TOPSONG&limit=10"
```