How the numbers are made

Most disagreements about social performance are not disagreements about performance. They are two people using the same word for different arithmetic.

A single luminous curve emerging from a dense field of scattered points

Metric dictionary

Each of these is defined once, in one place, and every figure on this site derives from that definition. The point is not that these are the only correct choices. It is that the choice is written down and does not drift between decks.

Engagement

Engagement rate, reach-based
Total engagements divided by total reach, aggregated across posts. The default.
Engagement rate, follower-based
Total engagements divided by follower count at period end. Roughly three times the reach-based figure. Reported separately and labelled, never mixed.
Aggregate, not averaged
Sum the numerators, sum the denominators, then divide. Averaging per-post rates lets a 200-reach post count as much as a 200,000-reach one.

Share of voice

Volume share
Brand mentions divided by all mentions in the category. Familiar and sentiment-blind. Kept because removing it starts an argument, but never shown alone.
Quality share
Volume weighted by net sentiment. The correction. The gap between this and volume share is the reportable finding.
Crisis severity
Modified z-score against a trailing 28-day median, excluding the day under test. A standard-deviation detector lets an incident inflate its own threshold and mask itself.

Content and testing

Format multiplier
Share of platform reach divided by share of platform calendar. Above 1.0 a format earns more reach than the slots it takes. Computed within platform only.
Practical significance
A band agreed before a test runs. Separates "ruled a difference out" from "learned nothing". Set per test against production cost.
Sample floor
Day-part cells below 20 posts render blank rather than ranked. A blank cell is information.

Taxonomy architecture

Listening platforms give you two ways to find themes, and they are good at different jobs. Using one for both is the most common way a listening programme becomes untrustworthy.

A curated Boolean taxonomy owns the recurring reported numbers. It runs on everything, it is auditable, and it stays stable week to week. Unsupervised discovery runs only on the residual the taxonomy could not place, and its output is next month's taxonomy entries rather than this month's report.

A model re-fit each week silently renumbers and reshapes its own topics, so "price mentions doubled" can mean the topic changed rather than the world did. For a figure in a recurring report, that is disqualifying. Equally, running discovery across the whole corpus just rediscovers the themes the rules already cover and buries the new signal underneath them.

Coverage, not accuracy, is the health metric. Taxonomy accuracy on placed mentions is close to meaningless, because the rules were written against that vocabulary, so it measures consistency. Coverage falling is the real signal. It means the conversation moved and the rules did not.

QA checklist

Every recurring report passes this before it leaves my hands.

  • Denominator stated. Every rate says what it is a rate of, in the label, not the footnote.
  • Window stated. Date range and comparison basis on the chart, not in the email body.
  • Sample size visible. Any cell or segment below the floor renders blank rather than ranked.
  • No pooled format comparisons. Formats compared within platform, because platform effects dwarf format effects.
  • Aggregate rates only. No mean-of-per-post-rates anywhere in the pipeline.
  • Significance separated from importance. A result carries an interval and is read against the practical threshold.
  • Nulls split in two. "Difference ruled out" and "test could not see" are reported as different findings.
  • Every headline number recomputable. If I cannot regenerate it from committed code on request, it does not ship.
  • Manual refreshes labelled as manual. Nothing is called live unless it is live.

The code behind the case studies

These are the working repositories, here for anyone who wants to check the arithmetic. The case studies are written to stand on their own without them.

How I stop share of voice rewarding the brand having its worst week

Explainable lexicon sentiment, a taxonomy-first theme architecture with discovery over the residual, quality-weighted share of voice, and a crisis detector whose baseline cannot be contaminated by the event it is testing.

22,319 mentions6 brands33 tests View source

How I tell "we ruled out a difference" apart from "the test learned nothing"

A/B evaluation with confidence intervals against a practical-significance threshold, peeking risk measured on the actual data rather than described, Wilson intervals on small segments, and creative fatigue fitted on log click-through against frequency.

$1.5M spend240 creatives36 tests View source

How I define an engagement rate so a 200-reach post can't outvote a 200,000-reach one

Twelve KPIs defined once, aggregate rates throughout, day-part timing with a 20-post floor and blanked thin cells, within-platform format comparison, and an auto-written stakeholder narrative.

2,092 posts4 platforms28 tests View source

A note on the data in all three

All synthetic, seeded and reproducible. No employer or client data appears in any of them. Each generator plants known events at known dates, which is what makes the pipelines testable against ground truth rather than merely plausible.