On the data
Every mention here is generated by a seeded script for six fictional beverage brands. No employer or client data appears anywhere in this project. That is not a compromise. The generator deliberately plants a recall, a launch spike and a trend the taxonomy has no rule for, at known dates, which means the pipeline can be tested against ground truth rather than admired. A real export could never support that.
Why it was hard
Three things make competitive listening harder than it looks, and each one quietly breaks a standard dashboard.
Volume is sentiment-blind. Share of voice is almost always reported as a share of mention count, which measures how much people are talking, not whether any of it helps. The failure mode is not subtle. It inverts precisely during the weeks a brand most needs an accurate read.
Sentiment without themes just relocates the work. "Sentiment is down four points" prompts exactly one question, which is about what? A number that cannot answer it generates a second analysis request instead of a decision.
A standard-deviation spike detector fails during the actual crisis. Flag any day more than three sigma above the mean and a genuine incident inflates the very mean and standard deviation used to judge it. By day two the threshold has climbed above the event. The crisis masks itself, which is the one moment the alert exists for.
Method
Sentiment runs on an explainable lexicon rather than a black-box classifier, because "the model said so" ends the meeting badly when a brand lead asks why a mention was scored negative. Every score can be decomposed into the terms that produced it.
Themes are handled by two mechanisms doing two different jobs. A curated keyword taxonomy owns the reporting, because a model re-fit each week silently renumbers and reshapes its own topics, so "price mentions doubled" can mean the topic changed rather than the world did. For a figure that goes into a recurring report, that is disqualifying. But rules only find themes somebody already thought of, which makes them blind to exactly what listening exists to catch. So NMF runs only on the mentions the taxonomy could not place. Run on everything, it would rediscover the covered themes and bury the new signal underneath them.
What it found
Two brands hold a tenth of the conversation and none of the goodwill. Volume alone hides it completely.
share_of_voice(), reproduces
on a clean checkout.
Solara holds 13.7% of all mentions and 0.0% of the favourable ones. Halcyon holds 10.1% and, again, 0.0%. On a volume dashboard both look like mid-table competitors worth planning against. Weight the same mentions by sentiment and the share they hold turns out to be entirely complaint.
The mirror image matters just as much. Verdant converts 16.5% of volume into 32.3% of favourable conversation, the largest positive gap in the set, while a volume ranking puts it fourth. The gap between the two dots is the finding, and a single-number chart cannot express it.
Severity separates a real incident from ordinary Tuesday noise. The recall scores 20.2 against a next-highest of 4.7.
test_baseline_excludes_the_day_under_test and
test_a_sustained_event_does_not_mask_itself.
The discovery half
The generator plants a "sunrise ritual" trend whose vocabulary appears in no taxonomy rule. It lands in the residual, and discovery surfaces it as the top emerging cluster at 1,679 mentions. That is the whole argument for the two-mechanism split, and it is asserted in a test rather than claimed in a README.
Measured, not assumed
A listening tool whose accuracy is unknown gets trusted exactly as much as one that has been measured, and wrongly. Both classifiers are scored against the labelled fixture, and tests pin floors so these figures cannot quietly drift into fiction.
Sentiment
- 97.1% polarity accuracy
- Of mentions that are truly polar and were called polar, the direction is right. It does not include the ones called neutral, and saying so is the point.
- 80.9% overall, three-class
- The stricter number, and the one to quote. Neutral is the hard class.
Taxonomy
- 85.1% coverage
- The health metric. A fall means the conversation moved and the rules did not.
- 100% accuracy on placed mentions
- Partly true by construction, because the rules were written against this vocabulary. A consistency check, not a generalisation claim.
What I would report
Volume share stays in the deck, because stakeholders know it and removing it starts an argument that is not worth having. It goes next to quality share, with the gap called out as its own number. The recurring line is not "we hold 10% of the conversation" but "we hold 10% of the conversation and 0% of the goodwill, and here is the theme driving it."
Crisis alerting moves off standard deviation and onto a trailing median with the day under test excluded. Everything else runs off the taxonomy so the numbers stay comparable week to week, and the residual gets reviewed monthly for entries the rules are missing.
Limitations, and what I would do differently
The sentiment lexicon is English-only and reads sarcasm badly, which in practice means a negative-leaning bias on exactly the platforms where irony is the register. That is a real weakness for a beverage brand on X or TikTok. Net sentiment is also a blunt weight: a mention at minus 0.9 and one at minus 0.1 both drag quality share down, and the second is often just mild disappointment.
Taxonomy accuracy at 100% is the number I trust least, and I have said why above rather than letting it sit in a table looking impressive. The honest test is coverage on conversation the rules were not written against.
Given more time I would weight quality share by engagement as well as sentiment, so a negative post seen by two hundred thousand people does not count the same as one seen by nine. I would also add a second annotator to the fixture. Every accuracy figure here is scored against labels from a single source, which measures consistency more than it measures truth.
Artefacts
Source and tests on GitHub, the metric definitions and Boolean taxonomy, and 33 tests passing on a clean checkout.
