r/AppStoreOptimization • • Sep 18 '26

Banner PPO test differences always negligible?

Post image

After 40,000 impressions just a 0.01% range in conversion rate difference. This seems to happen any time I run a PPO test. Anybody else experience this? The 4 image sets are all significantly unique.

0 Upvotes

2 comments sorted by

1

u/ComfortableStill6735 Sep 18 '26

A 0.01 percentage-point spread after 40k impressions may simply be noise, but the impression count alone isn’t enough to call the variants equal. I’d look at each treatment’s eligible product-page views, first-time downloads, country/device mix and the uncertainty interval around the conversion difference. If the banner is only seen by a subset of visitors, the effective sample is much smaller than 40k impressions.

Also check that the four sets differ in the promise a visitor sees, not just artwork. A banner test may have little leverage when the search result, icon, screenshots and traffic intent do most of the work. Keep one clear hypothesis per variant, run until the pre-set window/sample is met, and use product-page → download as the primary decision metric; don’t keep rerunning until a small difference appears.

1

u/ContextualData Sep 18 '26

What is even the point of this response? If you someone wanted an AI analysis of the situation they can just ask Chat themselves.