Before and after: how to read a listing change without a store test

Why two windows differ when nothing changed#

A before-and-after read compares the weeks after a change with the weeks before it. The trouble is that two runs of weeks differ anyway. Know the usual reasons before you read any change.

Ads need the most care on the App Store, because Apple counts them inside the source a keyword change aims at. Its definition of App Store search ends:

So an Apple Ads campaign that starts, stops or changes its bids during either window can move App Store search. Apple Ads can also run on the Today tab, the Search tab and product pages (Apple Ads glossary), and App Store Connect's source definitions mention ads only under App Store search. Keep ad spend steady across both windows, or call the read unclear. Play's store listing reports keep ads apart: Google's definitions leave visits from ads out of Google Play search and Google Play explore, and count them under Ads and referrals. The Grow overview does not: its Paid and direct source counts installs from ads together with searches for your app's name.

Choosing the windows#

1. Date the change by the day it appeared on the public listing, not the day you submitted or approved it. Check the store page and write the date down.
2. Compare whole weeks: 7, 14 or 28 days after the change against the same number of days just before it. Each window then holds every weekday equally often.
3. Leave out the day the change went live, since part of that day still saw the old listing.
4. Wait until the last day of each window is complete before you read it.
5. Keep everything else still if you can: no second listing change, new campaign or price change inside either window. If something else changed, say so in the readout.

Apple says when a day is complete, in its analytics reports documentation:

So a read that ends yesterday reads days that are not finished yet. On the App Store, end your windows at least two days back. On Google Play, leave the newest few days out as well, and export again before the final read.

Read the source the change was aimed at#

A change acts on one part of the funnel first. Read that part, by source, and use the rest as a check.

• Keyword field, title or subtitle on the App Store: App Store search impressions first, then App Store search downloads. More impressions at the same conversion rate is the cleanest read. If impressions rise and conversion falls, the new words may reach searches that fit the app less well.
• Icon, screenshots or preview video: the conversion rate, source by source. Apple's conversion rate is total downloads and pre-orders divided by unique device impressions.
• A description change on Google Play: Google Play explore, not Google Play search. In Play's store listing reports, Google Play search counts only searches for your app's name or a closely associated brand. Searches for a kind of app, such as "racing game", count under Google Play explore.
• Graphics on Google Play: the click-through rate, the share of store listing visitors who clicked Install, Open or Pre-register. An icon change can also move how many people visit, so read store listing visitors by source too.

Then check the sources the change should not touch. If App Store browse and web referrer moved by the same share as App Store search, the season or a feature is the likelier story. When you changed only one store's listing, the other store makes a second check.

If impressions rose but downloads did not follow, search impressions not becoming downloads walks through the usual reasons.

The smallest change a small app can see#

Chance alone moves small counts a lot. The sample-size formula that sizes a store test also tells you how big a change a before-and-after read could pick out, if weeks differed only by chance: visitors per side = (zα + zβ)² × (p1(1 − p1) + p2(1 − p2)) ÷ (p2 − p1)². At 95% confidence and 80% power, zα + zβ = 1.96 + 0.8416 = 2.8016, and 2.8016² = 7.8490.

An example with invented round numbers: an app gets 1,000 App Store search impressions a week and 50 downloads from them, a 5% conversion rate. Compare four weeks before with four weeks after, so 4,000 impressions on each side.

• A 30% change, from 5% to 6.5%, needs 7.8490 × (0.05 × 0.95 + 0.065 × 0.935) ÷ 0.015² = 7.8490 × 0.108275 ÷ 0.000225 = 3,778 impressions a side. Four weeks hold 4,000: just enough.
• A 20% change, from 5% to 6%, needs 7.8490 × 0.1039 ÷ 0.0001 = 8,156 a side: more than eight weeks on each side.
• A 10% change, from 5% to 5.5%, needs 31,232 a side: about 31 weeks on each side.

That is the floor, not the answer. Real weeks also differ for the reasons above, so the true smallest change is larger. Measure your own swing: take pairs of past four-week windows when you changed nothing, and see how far apart their conversion rates sit. If they often differ by 15%, a 10% move after a change says nothing either way.

The store test calculator shows the four-week floor for your own numbers, next to what a store test could show on the same traffic. It starts at 90% confidence; set it to 95% to match this table.

Reading at 7, 14 and 28 days#

Reading a change more than once makes sense, because a large problem shows early. But every look is one more chance for noise to clear your bar.

Say each read calls a change helpful when a rise that large would happen by chance only 1 time in 20: a one-sided bar, which looks only for a rise. Read once, and a change that did nothing gets called helpful 1 time in 20. Read at 7, 14 and 28 days and act on the first read that clears the bar, and it happens about 1 time in 9: more than double. A two-sided 95% bar, where a false call in either direction counts, goes from 5% to about 11% over the same three looks. Both figures assume days differ only by chance; real swings push them higher.

The fix is a stricter bar at the early looks, so the chances add up to what you meant. For example, bars of 1 in 200 at 7 days, 1 in 100 at 14 days and 7 in 200 at 28 days add up to 1 in 20, so the chance of a false call across all three looks stays at or below 1 in 20. The early looks then react only to a large move, and the 28-day look carries most of the weight.

The same goes for reading many numbers at once. Check ten unrelated numbers at a 1-in-20 bar and the chance that at least one looks significant by luck is about 40% (1 − 0.95¹⁰). Decide before the change which source and which number will judge it.

Writing an honest readout#

A readout says what followed the change, over which windows, from which source, and how sure anyone can be. It says it was not a controlled test. When the move sits inside the app's usual swing, it says unclear.

1. The change: the field, the store, the country and language, and the day it appeared on the listing. On the App Store, the app name and the subtitle change with a new app version, and the keyword field is part of each version's information, so name the version too.
2. The windows: the dates, equal lengths in whole weeks, and any days left out.
3. The numbers: the targeted source before and after, with its conversion rate, and the other sources as a check.
4. What else moved: ads, featuring, ratings, releases, rivals.
5. The read: up, down or unclear, against the app's own swing and the bar for this look.
6. The words: before and after, not a controlled test.

NoteExample readout, with invented numbers. Subtitle changed on the App Store (US, English), live on March 3. App Store search, 28 days on each side: impressions 9,800 then 11,100 (+13%), downloads 480 then 520 (+8%), conversion 4.9% then 4.7%. App Store browse and web referrer flat; Apple Ads spend unchanged. Past four-week windows differed by up to 12% in impressions. Read: unclear. The rise in impressions sits at the edge of this app's usual swing, and downloads moved less. Before and after, not a controlled test.

Leave out every verb that hands the change the credit for the move. A before-and-after read can say what followed a change. It cannot say why.

When a store test is the better tool#

If the change is to images or video and your traffic can show the size of change you care about, test it instead. A randomized test runs both versions over the same days, so the season and the ads land on both. On the App Store, product page optimization tests the icon, screenshots, app previews and the new header and search result assets shown on iOS 27 and later; on Google Play, store listing experiments also test the descriptions. The store listing tests guide covers both.

If your traffic is too small for either, make changes big enough to see, one at a time, and leave each live long enough to read.

Where RocketASO fits#

RocketASO reads your App Store and Google Play reports, names the fix estimated to be worth most this week, and compares installs before and after it. Each change is read 7 and 14 days after it goes live, against the same days before: a before-and-after read, not a controlled test. It starts you with a draft of the copy or the keyword field, and nothing reaches the store until you approve it. The free ASO audit checks a public listing without sign-up.