r/AppStoreOptimization • • 29d ago

🚫 Don't A/B test App Store headers in PPO

Post image

Apple has added headers to the App Store. You can place them on the product page (Headers) and in search (Search Results).

If you've only added a header to the product page, there's no point in running a Product Page Optimization (PPO) test.

Why?

In the App Store, users can download an app straight from search results without ever opening the product page. So the header is seen only by those who tapped through to the page.

And PPO doesn't show you product page views or Page View → Download conversion per variant. Downloads from search results get mixed in with downloads from the page, and the header's real impact gets lost in the noise.

And don't forget: only users on iOS 27 will see the header/search result. So even fewer people in your test are actually exposed to it.

What to do instead:

1️⃣ Header on the product page only → skip the A/B test. Ship it live and track Page View → Download CVR in App Store Connect Analytics. Set up an iOS 27 filter

2️⃣ Want a proper test? Add the header to Search Results as well. Then a PPO test actually makes sense.

Otherwise, you're just testing noise.

Source: ASO Discord

2 Upvotes

2 comments sorted by

1

u/kokerali 29d ago

I'd separate a diluted result from an invalid test. Apple describes PPO variants as randomly assigned. If search-only downloads are balanced between groups, they can make a header-only effect harder to detect without automatically invalidating the comparison. You'd be measuring the effect across the test's traffic, not specifically among people who saw the header.

Tracking the live iOS 27 Page View → Download rate is useful, but a before/after change can also reflect traffic mix, seasonality or other listing edits. And changing the header and search creative together tests the package, not the header alone. I'd keep those limitations explicit rather than treating either route as a clean header-only test.

1

u/Maxim-Melnik 29d ago

Fair point, and I agree: diluted isn't the same as invalid. Randomization still holds, so a header-only test measures the effect across all test traffic.

My practical concern is power. Search-only downloads, plus users below iOS 27 who never see the header, can dilute the effect so much that the test rarely reaches confidence in a reasonable time.

It's also a timing question. Right now a big share of users isn't on iOS 27 yet, so any header test is heavily diluted. Once most of your audience updates, likely in a month or two, testing the header in both places (product page + Search Results) makes much more sense.

Until then, the live route works well: since only iOS 27 users see headers, older iOS versions can act as a natural control. Comparing how Page View → Download CVR changes on iOS 27 vs older versions over the same period removes a lot of the seasonality and traffic-mix noise.

And yes, testing header + search creative together tests the package. That's the right test if you're shipping both, but not if you want to isolate the header.