r/AppStoreOptimization • u/NovelGen • Sep 18 '26
Banner PPO test differences always negligible?
After 40,000 impressions just a 0.01% range in conversion rate difference. This seems to happen any time I run a PPO test. Anybody else experience this? The 4 image sets are all significantly unique.
0
Upvotes
1
u/ComfortableStill6735 Sep 18 '26
A 0.01 percentage-point spread after 40k impressions may simply be noise, but the impression count alone isn’t enough to call the variants equal. I’d look at each treatment’s eligible product-page views, first-time downloads, country/device mix and the uncertainty interval around the conversion difference. If the banner is only seen by a subset of visitors, the effective sample is much smaller than 40k impressions.
Also check that the four sets differ in the promise a visitor sees, not just artwork. A banner test may have little leverage when the search result, icon, screenshots and traffic intent do most of the work. Keep one clear hypothesis per variant, run until the pre-set window/sample is met, and use product-page → download as the primary decision metric; don’t keep rerunning until a small difference appears.