Incrementality testing on a small budget with geo holdouts

Platform dashboards tell you which conversions a channel claimed. They do not tell you which conversions would have happened anyway. With attribution already patchy on iOS, that gap matters more than it used to. The cleanest way to close it is an incrementality test, and a geographic holdout is the version a small team can run without special platform access.

What a geo holdout does

The idea is simple. You pick some regions where advertising continues (test) and some where you switch it off or hold it back (control), then compare outcomes over the same period. The difference, adjusted for how the regions behaved before the test, is your estimate of what the advertising added.

Meta's Marketing Science team publishes an open-source package for this called GeoLift. Its documentation positions it for situations where people-based measurement such as Conversion Lift is not feasible, and notes that Meta recommends people-based experiments when possible because they have higher statistical power. For a small advertiser, geo is often the practical choice anyway, because it works with your own sales or install data and does not depend on any one platform's reporting.

Pick the outcome from your own data

GeoLift's walkthrough expects a simple table: a location, a date and an outcome. Use a number you trust and can break down by region. For an ecommerce app, that might be orders from your own backend. For a subscription app, it might be trial starts. Avoid platform-reported conversions as the outcome, since they are the thing you are trying to check.

Choose regions with the power question in mind

The GeoLift workflow includes market selection and power analysis: it simulates many combinations of test and control markets and ranks them by statistical power and minimum detectable effect. You do not have to use the package to borrow the logic. Pick regions whose history moves together, avoid very small ones where a single large customer swings the numbers, and make sure the test regions are not systematically different from the control regions in ways that change with the season.

Here is the uncomfortable part for small budgets. Suppose you spend $20,000 a month in a single channel and run a test in half your regions. A modest real lift of, say, 5 percent in total conversions will often be indistinguishable from week-to-week noise. These round numbers are hypothetical, but the principle is general: a small budget buys the ability to detect a large effect, not a subtle one. If the channel is genuinely driving a third of your conversions, a geo test will see it. If it contributes a few percent, it probably will not, and you should say so before you start rather than after.

Set the length and the cooldown

The GeoLift walkthrough advises that the test be long enough to cover at least one full purchase cycle for your product, and notes that a post-campaign cooldown can be useful depending on the vertical. If people typically take ten days to convert after first seeing an ad, a one-week test mostly measures noise. Our default is four to six weeks of spend, with at least as much history collected beforehand to check that the regions track each other.

Run it cleanly

• Change one thing. Do not launch a new campaign structure or a discount in the same window.
• Use the platform's own location settings. Exclude the control regions at the campaign level so no budget leaks into them, and check delivery by region weekly.
• Hold other channels steady. If email or organic promotions spike in only some regions, your comparison breaks.
• Write down the hypothesis and threshold first. For example, "If the lift is below the cost of the media, we cut spend."

Read the result with humility

Report a range, not a point. A result of "between 5 and 25 percent incremental" is useful: it still supports a decision, and it keeps you from pretending to precision the data cannot give. If the range is too wide to act on, lengthen the test, test a bigger share of regions, or combine several weeks of spend on one channel rather than spreading thin across three.

Also remember what a single test shows. It describes this channel, in these regions, in this period. It is a calibration point for your platform-reported numbers, not a permanent correction factor. Run it again when the creative, offer or spend level changes materially.

The practical takeaway: pick one channel where you suspect the dashboard flatters you, run one properly sized geo test, and use the answer to adjust how much weight you give that channel's reported results.

Sources

• GeoLift documentation, Introduction
• GeoLift documentation, Walkthrough