How to A/B test your listing on the App Store and Google Play

What only a store test can tell you#

Compare this month with last month and you compare two different months. Holidays, a feature on the Today tab, a new ad campaign or a rival's new icon all differ between them. A store test splits visitors at random over the same days instead, so all of that lands on each version alike.

That is why, of the reads an app owner can run, only a randomized store test can speak to cause. A before-and-after read says what followed a change; the before-and-after guide covers how to do that one well. Both stores now run tests inside their own consoles. What they test differs, and so does how they report a result.

Product page optimization on the App Store#

Product page optimization shows up to three treatments of your product page to a share of App Store visitors, chosen at random, against your original page. Each person sees the same treatment for the whole test. Apple notes that treatment screenshots, previews and icons may also appear in search results and on the Today, Games and Apps tabs, so a test covers more than the product page. On iOS 27 and iPadOS 27 and later, the App Store also shows creative assets: a product page header and a search result asset. Apple says you can use product page optimization to test alternative header visuals.

1. In App Store Connect, open the app, click Product Page Optimization in the sidebar and create a test. Give it a reference name of up to 64 characters.
2. Choose one to three treatments and a traffic proportion: the share of visitors shown a treatment instead of your page. Apple's example: at 40% with two treatments, each treatment gets 20% of traffic and your original page the other 60%.
3. Choose the localizations. All of them are in by default, and a visitor shown a localization you left out is not in the test.
4. Optionally open Estimate Your Test Duration and pick the improvement you hope to see. Apple uses your existing data, such as daily impressions and new downloads, to estimate the days and impressions needed. The estimate does not change the test.
5. Set up each treatment: the app icon, then the Header and Search Results or App Previews and Screenshots tab (configure test treatments). Anything you leave alone shows your original page's version.
6. Start the test. If new assets need App Review, you add the test to a submission first (run a test).

• Icons: an alternate icon must already be in the binary of the app version on the App Store, built with Xcode 13 or later with alternate icons in the asset catalog, at 1024x1024 pixels. Someone who downloads from that treatment sees its icon on their device.
• Review: new screenshots or previews need approval, but not a new app version. Reordering screenshots already on the App Store, or only changing the icon, needs no resubmission.
• Who and where: treatments show on iOS 15 and iPadOS 15 or later, the app must be Ready for Distribution, and tests are not available for custom product pages (overview).
• Length: a test runs for 90 days or until you stop it, and Apple says you cannot change a test once it has started. A stopped test can't be restarted; you create a new one.

NoteApple's pages don't all agree. The Analytics help page lists descriptions among the elements a test covers, and says separate tests of different elements can run at the same time. The App Store Connect set-up pages name the icon, screenshots and previews, plus a Header and Search Results tab, but not descriptions, and the product page optimization page says you can create one test at a time. Plan on one test of images and video, and check what App Store Connect lets you create.

Store listing experiments on Google Play#

Play Console's store listing experiments compare up to two variants with your current listing. You can run them on your default listing and on custom store listings. There are two types, and per app you can run one default graphics experiment or up to five localized ones at the same time:

• Default graphics: the icon, feature graphic and screenshots, in your default listing language.
• Localized: the icon, feature graphic, screenshots and the descriptions, in up to five languages.

One trap: if you have added localized graphics in a language, people viewing your listing in that language are left out of default graphics experiments, even when you only test the icon. Google's example is an English listing with a French feature graphic: French viewers are excluded.

1. Go to Grow users > Store presence > Store listings, and click Set up in the Experiment column of the listing.
2. Name the experiment (50 characters or fewer) and pick the type, and for a localized one the languages.
3. Pick the target metric: unique user install clicks, unique user open clicks or unique user pre-registration clicks.
4. Review the advanced settings: the number of variants (up to 2), the audience percentage (the share of listing impressions shown a variant, split equally between variants), the minimum detectable effect and the confidence level.
5. Choose the asset type, add the variants, save, and submit the changes for review from the publishing overview.

Google says the default settings give results in the shortest time with reasonable accuracy, and that changing them can make the experiment run longer. The minimum detectable effect is the smallest difference that counts:

Experiments stop on their own after 6 months. After that no new data is collected, no variant is applied, and all traffic goes back to the current listing.

The two tests side by side#

How much traffic a test needs#

A test can tell two versions apart only once the gap between them is bigger than the noise. Noise shrinks slowly: to see a change half as big, you need about four times as many visitors. So the first question is not what to test, but how big a change your traffic can show.

The standard sample size for comparing two rates is: visitors per version = (zα + zβ)² × (p1(1 − p1) + p2(1 − p2)) ÷ (p2 − p1)². Here p1 is today's conversion rate and p2 the rate you hope to detect. zα sets the bar against chance: 1.6449 for 90% confidence and 1.96 for 95%, both two-sided. zβ = 0.8416 gives 80% power, so a real change of that size shows four times in five.

An example with invented round numbers: 1,000 unique impressions a day on the App Store and 50 downloads, a 5% conversion rate. One treatment at a 50% traffic proportion, so each version gets 500 impressions a day.

1. Goal: see a 20% relative change, from 5% to 6%. So p1 = 0.05, p2 = 0.06, and the gap is 0.01.
2. Spread: 0.05 × 0.95 + 0.06 × 0.94 = 0.0475 + 0.0564 = 0.1039.
3. Bar at 90% confidence: (1.6449 + 0.8416)² = 2.4865² = 6.1827.
4. Visitors per version: 6.1827 × 0.1039 ÷ 0.01² = 0.64238 ÷ 0.0001 = 6,424 unique impressions.
5. Days: 6,424 ÷ 500 = 12.8, so 13 days. Round up to 14 to keep whole weeks.
6. At 95% confidence the bar is (1.96 + 0.8416)² = 2.8016² = 7.8490, so 7.8490 × 0.1039 ÷ 0.0001 = 8,156 per version: 17 days, or three weeks.

Now aim for a 10% change, 5% to 5.5%. The spread is 0.0475 + 0.051975 = 0.099475 and the gap squared is 0.000025, so each version needs 6.1827 × 0.099475 ÷ 0.000025 = 24,601 impressions: 50 days at 500 a day. Half the change, nearly four times the visitors.

Round every figure up to whole weeks, and never run less than one full week, which is also Google's advice for covering weekday and weekend traffic. The top row is the small-app reality: a 10% change is out of reach of Apple's 90 days. When a test is unlikely to settle within 90 days, Apple may mark the variant Likely to be Inconclusive. When its own estimate runs past 90 days, Apple suggests fewer treatments or more of your traffic in the test. One more caution: Apple counts unique devices over the whole test, and people come back on other days, so a sum of daily unique impressions overstates what a test collects. Read the days in the table as a floor.

The store test calculator runs this math on a week of your own numbers: impressions and downloads on the App Store, store listing visitors and installs on Google Play. It splits its share of visitors evenly across every version, yours included, so Apple's 50% traffic proportion with one treatment is the calculator's 100% with two versions. It also shows the smallest change your traffic can show within 90 days.

NoteBoth consoles estimate the length too, each with its own method, and Apple reads results with Bayesian methods rather than this formula. Treat any estimate as a guide to weeks, not a date.

Reading the result and applying it#

On the App Store, open Analytics, then Product Pages under Acquisition. Each variant shows unique impressions, conversion rate, improvement over the baseline and a confidence figure (Apple). Results appear once the test has five first-time downloads and update daily.

Apple defines the confidence level as the probability that the test data suggest two variants perform differently, and works it out with Bayesian methods. Its estimated conversion rate incorporates the data observed during the test as well as existing data, so early estimates lean partly on your page's past. Apple gives a Performing Better label at 90% confidence or more, which by Apple's own measure leaves up to a 1-in-10 chance that the treatment is not really better. That is good evidence, not proof. Apple also shows a 90% credible interval for the lift. A wide one, say −5% to +40%, means the test has not said much yet.

On Google Play, the Results section explains the outcome from its confidence interval and your minimum detectable effect. It can name a variant that did better, call a draw, or say More data needed. Google defines the confidence level as how often the interval contains the listing's true performance; raising it lowers the chance of a false positive.

To apply an App Store treatment, open the test and click Apply Treatment to Current Product Page. It can go to your live page and to versions in Ready for Distribution or Prepare for Submission. Apple says only the screenshots and previews are applied; to use a treatment's icon, set it as the default icon in your next app version. Applying stops a running test.

On Google Play, apply the variant Play recommends, choose between several that did better, or click Keep current listing if yours did best.

Mistakes that produce a false winner#

• Stopping on the first good day. Results update daily, and early on the confidence figure swings. If you check daily and stop the first time a treatment crosses 90%, chance gets many tries, so a false Performing Better becomes more likely. Apple's help says not to end tests early, and also suggests applying a treatment once it shows Performing Better: do that at the end of the length you set, if the label still holds.
• Testing several things in one treatment. Change the icon and the first screenshot together and the result can't say which one mattered. Google recommends one asset at a time, and Apple suggests watching how many elements each treatment changes.
• Many treatments on little traffic. Each treatment takes visitors from the others, and each is another chance for a fluke to look like a win. On a small app, test one treatment.
• Ignoring the calendar. Randomizing protects the comparison, not the conclusion: a test run over a holiday tells you what holiday visitors preferred. Google advises revisiting assets for changes in users, locations and seasonality.
• Releasing a version mid-test. Apple warns that a new version may impact the results if it contains assets or metadata the test is testing (overview).
• Reading no result as a tie. More data needed or Likely to be Inconclusive means your traffic could not tell the versions apart. A draw on Play means no difference as large as your minimum detectable effect. Neither means the versions perform the same.

Where RocketASO fits#

RocketASO reads your App Store and Google Play reports, names the fix estimated to be worth most this week, and compares installs before and after it, 7 and 14 days after the change goes live: a before-and-after read, not a controlled test. It starts you with a draft of the copy or the keyword field, and nothing reaches the store until you approve it. To see where a listing stands first, run the free ASO audit.