On the data
Seeded, reproducible, synthetic. No employer or client data appears anywhere in this project. Each A/B test is generated with a known true lift, including one genuine null, so the statistics can be checked against ground truth rather than trusted. Every observed interval below contains its planted value.
Why it was hard
Paid social measurement is not hard because the maths is difficult. It is hard because the wrong answer is indistinguishable from the right one on the surface, and the person reading the deck has no way to tell them apart.
Stopping the test the day it looks significant
Checking a running test daily and stopping at the first p < 0.05 does not run a 5%
false-positive test. Every extra look is another chance for noise to cross the line, and the
effective error rate climbs with the number of looks. This is the single most common way a paid
social test produces a confident, wrong answer.
Rather than describe the risk, the analysis measures it on the actual test data, recomputing the p-value after each day and flagging the dangerous case explicitly: significant partway through, not significant at the end. That is a test an analyst watching daily would have shipped, and it is visible only if you look for it deliberately.
Reporting a p-value with no interval
"B wins, p=0.03" is not something a budget can move on. A 2% lift and a 40% lift can carry the same p-value at different sample sizes and imply opposite decisions. Every verdict here carries a confidence interval and is read against a practical-significance threshold of plus or minus 10%, because a 1% improvement in click-through can be entirely real and still not justify re-shooting the creative.
Reading "not significant" as "no difference"
This is the one that quietly costs the most. A flat result on a thin test means we learned nothing, not the variants are equal, but it gets filed as a win for the control and shapes strategy for a year.
What it found
AB-03 and AB-04 are both "not significant" and they are not the same result. One ruled a difference out, the other could not see anything.
Called
- AB-01 Hook, social proof against founder story
- Observed +20.6% with an interval of +13.8 to +27.3, against a planted +22%. Variant B wins.
- AB-02 Format, short video against static
- Observed +33.8% with an interval of +25.3 to +42.2, against a planted +41%. Variant B wins.
Not called, for two different reasons
- AB-03 Copy, discount-led against value-led
- Observed minus 2.5% with an interval of minus 9.2 to plus 4.2, against a planted 0%. The interval sits entirely inside the threshold, so a difference worth acting on has been ruled out. The copy debate can stop.
- AB-04 Landing page, long against short
- Observed +6.6% with an interval of minus 10.8 to plus 23.9. The interval still contains a +24% effect, so the test could not see anything. The question is open.
The interval does this job, not post-hoc power. Observed power is a deterministic function of the p-value, so it adds nothing the test has not already said and labels every null as underpowered. It is reported for reference and no verdict depends on it.
Creative fatigue
The most expensive thing in a paid account that nobody has an alert for. Click-through decays as the same audience sees the same ad repeatedly, CPMs climb as the auction saturates, and spend continues at a steadily worsening rate until someone notices in a monthly review.
Fatiguing creatives are under half the account and carry three-fifths of the spend.
log(CTR) against frequency, so the slope reads as a proportional
change per unit of frequency and stays comparable across creatives with very different baseline
click rates. A raw linear slope does not. 202 creatives had enough delivery to analyse.
What I would report
Test readouts carry three things and never fewer: the observed effect, its interval, and the practical-significance threshold agreed before the test ran. Verdicts use four labels rather than two, because collapsing "no meaningful difference" into "inconclusive" is the error that costs the most.
Fatigue gets an alert rather than a monthly discovery. Any creative whose fitted decay is significant and whose spend share is above a threshold surfaces weekly. And no test is called early. If a test must be monitored, the monitoring is recorded so the peeking cost can be stated rather than ignored.
Limitations, and what I would do differently
These are two-arm tests on a single conversion metric. Real paid social rarely stays that clean. Multi-arm creative rotations, overlapping audiences and platform-side optimisation all break the independence these intervals assume. Meta's own delivery optimisation actively works against a clean split, which is a measurement problem no amount of care at analysis time fixes.
The threshold is a judgement, not a fact. It should be set per test against production cost, because re-shooting a video justifies a higher bar than swapping headline copy. Using one number for everything is a simplification I would not defend in a real account.
Given more time I would add sequential testing with an alpha-spending function, which lets a test be monitored honestly instead of being told not to look. Telling analysts not to peek does not work. Giving them a valid way to peek does.
Artefacts
Source and tests on GitHub, the metric definitions and QA checklist, and 36 tests passing on a clean checkout.
