Amazon A/B test calculator: is your listing change a real win?
Free Amazon A/B test calculator: significance, probability that B wins, the range of the real lift and how many sessions you still need. Experiments or before/after.
Gazi El Halabi ·
In short
Enter sessions and units ordered (or impressions and clicks) for both versions. The calculator tells you whether the difference is real or noise: lift, probability that B is better, p-value, the 95% range of the true lift and how many more days you need. It also warns you when a before/after comparison is being skewed by ads, seasonality or price. It runs in your browser.
Promising, not proven
The data leans one way but the sample is still too small to call it. Keep the test running before you publish anything.
Lift of B over A
+15.6%
Probability B is better
98%
p-value
0.043
Sample needed per version
4,783
≈ 14 more days at your current traffic
95% range of the true lift
Example data loaded — replace it with yours. Frequentist: two-sided two-proportion z-test at 95%, power 80%. Bayesian: Beta posteriors with a flat prior. Runs in your browser; nothing is sent or stored.
How to read the result
| Output | What it tells you |
|---|---|
| Lift | How much better (or worse) B performed than A, relative to A |
| Probability B is better | How likely it is that B's true rate is higher — the kind of probability Manage Your Experiments reports |
| p-value | How surprising the difference would be if A and B were really the same; under 0.05 is the usual bar |
| 95% range of the true lift | The plausible range of the real effect. If it crosses zero, you have not proven anything yet |
| Sample needed per version | The traffic needed to detect the lift you see with 80% power |
A result is only a winner when the difference is significant *and* the sample is large enough. Stopping a test the first day it looks good is the most common way to publish a change that does nothing.
Why before/after comparisons lie
When you compare last month with this month, your change is mixed with everything else that moved: seasonality, price, coupons, ad budget, stock, reviews and what competitors did. Amazon's Manage Your Experiments avoids that by splitting shoppers into two groups at the same time — one group sees version A, the other sees version B, for the whole experiment. Whenever the brand is eligible, that is the comparison to trust.
If you have to compare periods, match them (same length, same days of the week, no sale event in only one of them) and check that ad spend did not jump: otherwise part of the lift is paid traffic, not the listing.
How much traffic a listing test needs
Sessions needed per version to detect a lift in conversion rate (two-sided 95%, power 80%):
| Current conversion | +5% lift | +10% lift | +20% lift | +30% lift |
|---|---|---|---|---|
| 5% | 122,124 | 31,234 | 8,158 | 3,780 |
| 10% | 57,763 | 14,751 | 3,841 | 1,774 |
| 15% | 36,310 | 9,257 | 2,402 | 1,106 |
The lesson in that table: small tweaks need enormous traffic to prove. On most listings, a 5% improvement is invisible within a month. Test bold, meaningful changes — a new main image angle, a rewritten title — not a single word.
CTR tests are even hungrier, because click-through rates are small: at a 0.5% CTR, detecting a 20% lift needs about 85,862 impressions per version.
Read CTR and conversion together
A main image that wins more clicks can still lose sales if it attracts the wrong shoppers or promises something the listing does not deliver. I always look at both: CTR to see whether the change earns attention, conversion (unit session percentage) to see whether that attention buys. If you can, look at units per impression — it captures both in one number.
My rules for listing experiments
- Write the hypothesis and the success metric before the test starts.
- Change one thing per experiment.
- Run full weeks — at least one, ideally two or more — because weekdays and weekends convert differently.
- Don't peek and stop early when it looks good; decide the sample size up front.
- Prefer bold changes: they are the ones you can actually measure.
Not sure what to test first? The main image test shows the patterns I check, and a free mini-audit points to the biggest opportunity on one listing.
Sources
Need the whole catalogue checked, not one title?
Request a catalogue audit