Four button versions, one question
Eniac's "SHOP NOW" button was tested in four variants across roughly 100,000 visits over two weeks, tracking click-through rate (CTR) as the primary metric.
Two independent variables โ button color (original vs. red) and button copy ("SHOP NOW" vs. "SEE DEALS") โ were combined into four versions. That's a 2ร2 factorial design: a multivariate test (MVT) wearing an A/B test's clothes. The difference matters because an MVT can reveal interaction effects โ cases where two changes don't just add together, but actively work against each other. A simple A/B test would never catch that.
Key design questions before launching
Before touching the data, we had to answer a handful of design questions โ the kind that determine whether the results mean anything at all.
What the test actually showed
The differences are real, not noise
Copy wins, color backfires
Changing only the text to "SEE DEALS" (Version C) nudged CTR slightly above control. Changing only the color to red (Version B) โ the "obvious" high-contrast move โ actually made things worse.
Combining both changes made things worse, not better
Version D โ red color and new copy โ didn't land between B and C. It scored lowest of all four versions, well below even Version B alone.
Pairwise comparisons, Bonferroni-corrected
To see exactly which versions differ from which, we ran pairwise chi-square tests with a Bonferroni-corrected threshold (ฮฑ = 0.05 / 6 = 0.00833) to avoid false positives from multiple comparisons.
| Comparison | P-value | Significant? |
|---|---|---|
| A vs B | 7.20ร10โปยนโต | โ Significant |
| A vs C | 0.474 | โ Not significant |
| A vs D | 2.28ร10โปยณยฒ | โ Significant |
| B vs C | 2.33ร10โปยนโท | โ Significant |
| B vs D | 2.82ร10โปโต | โ Not significant* |
| C vs D | 6.48ร10โปยณโถ | โ Significant |
*B vs D misses the Bonferroni-corrected threshold, even though the raw p-value looks small โ a good reminder of why the correction matters when running six comparisons at once.