Every media dollar eventually has to answer the same uncomfortable question: Did it create new revenue or simply take credit for revenue that was coming anyway?
A channel with exceptional reported ROI may simply be capturing existing demand. Another may appear less efficient while creating customers who otherwise would never have entered the market. That distinction matters more than which platform receives credit for the final conversion.
A new generation of off-the-shelf automated Marketing Mix Modeling (MMM)* platforms has made it faster and easier to answer that question.
But that accessibility introduces a new risk: mistaking a polished answer for a definitive one.
In a head-to-head comparison of several MMM approaches, we found that the same business data could produce meaningfully different conclusions. Data preparation and model specifications changed the estimated contribution of individual channels. When signals conflicted, a model could identify a pattern without explaining why it existed.
A model can also evaluate only the variables it is given. If growth drivers such as pricing, competitor dynamics, distribution, seasonality, customer experience or organic demand sit outside the data set, it may overattribute performance to the media variables it can see.
MMM’s next challenge, therefore, is developing a more rigorous way to challenge, validate and apply its conclusions.
What has actually improved
The traditional criticism of MMM was its speed and rigidity. Custom models could take months to build and be revisited only once a year at most.
Today’s platforms can ingest data more efficiently, refresh models regularly and incorporate industry benchmarks. Their scenario-planning and visualization capabilities have also made MMM more useful in executive budget conversations. A diminishing-returns scenario is far more actionable to a CMO than a table of statistical coefficients.
Some platforms are also combining methodologies. Geo holdouts and conversion-lift experiments can calibrate or challenge modeled estimates rather than living in separate reports.
That combination moves marketers beyond relationships observed in historical data toward conclusions tested against something closer to causation. But a holdout is only as credible as its design, execution and interpretation.
Plausible, but not always useful
An MMM can fit historical data well and still lead to the wrong business decision. Statistical fit indicates how well a model explains the past. It does not establish that the model has correctly identified causation or that reallocating future investment based on its findings will generate growth.
This becomes difficult when variables move together. Television may run alongside paid search, social alongside promotions and strong fourth-quarter sales alongside holiday demand. Earned attention may also increase branded demand across several channels simultaneously.
The model must separate these effects using incomplete data and limited real-world variation. Its assumptions matter: How quickly does an advertising effect decay? How long does a channel take to produce an outcome? How much should benchmarks influence the estimate? What belongs in the business baseline rather than marketing’s incremental contribution?
These settings can materially change which channels appear effective and where the model recommends investing the next dollar.
Where MMM still breaks down
In one off-the-shelf model we evaluated, the platform estimated that 10 paid-media channels drove approximately 75% of new-customer volume. That immediately raised questions because it was significantly higher than anything we had observed in the business. To date, I have seen models identify 1%-18% incrementality from paid media.
We then gave the same underlying data to a data scientist for an independent custom build. That model estimated the combined incremental contribution of those channels at less than 1%.
The point is not that the custom model was automatically correct. A result below 1% deserves as much scrutiny as a result of 75%. But a gap of that magnitude makes either conclusion impossible to treat as decision-ready without further investigation.
That investigation revealed that the client’s media variables had limited ability to explain changes in the outcome that the models were attempting to predict.
When client data provides a weak or ambiguous signal, a Bayesian MMM may remain heavily influenced by its prior assumptions from other brands or benchmarks. These priors can stabilize a model when data is sparse, but they can also shape the answer more heavily than marketers realize.
Marketers need to understand when a conclusion is driven primarily by their own business data. Otherwise, a company could make major allocation decisions based on an industrywide expectation that does not reflect its individual business.
Data-hygiene issues and small specification changes could also materially alter which channels appeared to work. When model runs disagreed, the software could produce both answers without explaining which was more credible.
The risk is that a platform can return a statistically plausible allocation of contribution even when the evidence separating one channel’s impact from another is weak.
That risk grows when an analysis is limited mainly to paid media. If owned, earned, operational and market-level drivers are absent, marketers may debate how to divide contribution among the variables included in the model while overlooking the variables that actually explain the business.
This is where experienced practitioners remain essential: testing specifications, diagnosing conflicts, incorporating business knowledge and challenging recommendations.
The best of both worlds
Off-the-shelf MMM platforms can be effective, and not every company needs a fully custom model. The strongest approach often combines the speed and structure of a platform with the flexibility to adapt the model to the business. That takes people who understand how the business works and can question the model’s assumptions, inputs and results.
Platforms can accelerate data ingestion, model development and scenario planning. Custom analysis can test alternative specifications and investigate implausible results. Business teams can add context about pricing, distribution and operational changes. Controlled experiments can then test whether modeled relationships produce genuinely incremental behavior.
The objective is not to select the model that produces the most favorable answer—or even the strongest statistical fit. It is to determine which conclusions remain credible when the data, assumptions, business reality and resulting recommendations are all challenged.
MMM is becoming faster, more accessible and more sophisticated. That is real progress. But the smartest model is still not a substitute for knowing when its answer should not be trusted.
*We have moved away from Media Mix Modeling, as most models include all marketing variables (promotions, pricing, etc).
“Data-Driven Thinking” is written by members of the media community and contains fresh ideas on the digital revolution in media.
Follow Boathouse and AdExchanger on LinkedIn.
For more articles featuring Sonia Chung, click here.
