Case study · Multi-channel · Analytics
Checking a marketing mix model with a geo holdout
A mix model credits Meta with a 1.8 return, but a four-week geo holdout measures 1.2. Calibrating the model changes the budget split and adds an estimated 3% incremental revenue.
Illustrative scenario. The company and all figures are hypothetical and show how we’d approach the problem. This isn’t a client result, and your numbers will differ.
- 1.8
- Model's Meta return, before
- 1.2
- Geo holdout result
- 1.3
- Model's Meta return, calibrated
- +3%
- Estimated added incremental revenue
All figures are hypothetical.
The situation
Imagine a company spending $100,000 a month across Meta, Apple and Google, and using an open source marketing mix model (MMM) to plan budgets. The model says Meta returns 1.8 for every dollar, so it recommends giving Meta half the budget. But Meta's spend has always risen during seasonal peaks, and the team suspects the model is giving Meta credit for that seasonal demand.
The test
A geo holdout can check that claim directly. We'd turn Meta off in a matched 20% of regions for four weeks and compare revenue with the regions where it kept running. In this scenario, the test measures a return of 1.2, with a 95% confidence interval of 0.9 to 1.5. The model's 1.8 sits well outside that range.
Calibrating the model
The test result doesn't replace the model. A holdout measures one channel, at one time, at one spend level. The model covers every channel and helps with planning. So we'd feed the holdout into the model as a prior for Meta's return, which is an option that open source tools such as Meridian and Robyn support in different ways, and then run it again. The calibrated estimate for Meta comes out at 1.3.
What would change
| Channel | Return used | Spend before | Incremental revenue | Spend after | Incremental revenue |
|---|---|---|---|---|---|
| Meta | 1.2 | $50,000 | $60,000 | $40,000 | $48,000 |
| Apple | 2.0 | $20,000 | $40,000 | $25,000 | $50,000 |
| 1.3 | $30,000 | $39,000 | $35,000 | $45,500 | |
| Total | $100,000 | $139,000 | $100,000 | $143,500 |
This table simplifies things by assuming each channel's return stays constant as its spend changes. In reality, returns usually fall as spend rises, so a move like this should happen in steps, with each step checked before the next. The estimated gain of about 3% is modest, but it applies every month, and it's based on a measurement rather than a correlation.
The takeaway
Treat a mix model as a hypothesis until an experiment has tested its biggest claims. Start with the channel where the model's estimate would move the most money. Then retest periodically, because a calibration ages as creative, competition and spend levels change.