Julian Runge is speaking at AdExchanger’s Programmatic IO conference in New York City on September 28-29. If you haven’t grabbed your ticket yet, you can secure your seat here.
Marketing mix modeling isn’t a new idea. It’s a very old one, actually, which makes its current reinvention over the past few years all the more notable.
But there’s a related development afoot.
Open-source MMM (OS-MMM) packages, like Google’s Meridian, Meta’s Robyn and PyMC-Marketing, which is a Python library for Bayesian marketing analytics, have been gaining ground as privacy changes made attribution less reliable. The code they use is publicly available, inspectable and reusable.
That’s a meaningful shift all on its own. But add in agentic AI, and MMM – long the domain of data scientists and specialized consultants – becomes something almost anyone can do.
“Even if you have very little modeling background, it’s almost from zero to one,” said Julian Runge, an assistant professor of marketing at Northwestern University. “That’s an increase of infinity, in a way.”
But just because the barrier to entry is dropping considerably doesn’t mean you can skip the supervision. “You still need human intelligence that can understand what’s happening,” Runge said.
Runge, who co-authored the first-ever academic paper on open-source measurement, which was published earlier this year, spoke with AdExchanger.
Spoiler: He thinks marketing measurement might actually be solved. (Really, not really.)
AdExchanger: The trend toward open-source measurement is an important development, but OS-MMM, really? Do we need another acronym? 😭
JULIAN RUNGE: It’s just easier to say “open-source marketing mix modeling,” I guess! Or, even worse, “open-source media and marketing mix modeling,” as in “OS-m/MMM” with a lower-case “m.”
That was used in the blue ribbon panel report about the future of MMM published in 2023 by the Marketing Science Institute.
Now I know who to blame, lol. But seriously, what do marketers need to know about OS-MMM, and why are we starting to hear it come up more?
It’s slowly becoming mainstream, and the main driver is privacy: GDPR, COPPA, Apple’s deprecation of the IDFA. All of these things have made attribution unreliable and forced people to find something else. The big platforms saw this coming. Meta put out Robyn in 2021, Google and PyMC followed and then Google later updated its LightweightMMM package with Meridian.
But the idea was also to democratize access. Many large brands have been doing MMM for decades, but digital-first advertisers had no familiarity with it at all.
What are the main differences between Robyn, Meridian and PyMC?
They’re all open source, so the code is fully transparent, and you can inspect it. There’s no hidden bias.
The main differences are in the approach and ease of use. Robyn is the easiest. It takes a frequentist modeling approach [which are models built on observed data without prior assumptions], while Meridian and PyMC both use a Bayesian hierarchical framework [which are models that incorporate prior knowledge and get updated as new data comes in].
PyMC has the highest degree of customizability, but it also requires the most expertise to use.
Not to pop on my tinfoil hat, but is there really no bias in Robyn or Meridian? They were both developed by walled gardens.
I’m not aware of any component that would ex ante favor social or search advertising. That said, Meridian does lend itself more toward view-based and impression-based inputs, which reflects Google’s world, and Robyn leans more toward spend and sales, which reflects Meta’s direct-response approach.
It’s less about bias and more about orientation.
Fair enough. So let’s talk about the agentic angle. Can you actually just prompt an AI agent to run OS-MMM on real data?
Yes, actually, and that’s what makes it fascinating. You can chat with whatever agent you use – Claude, ChatGPT, Gemini – and tell it to pull the package, load the data and estimate the model. Since it’s open source, an agent can pull everything into memory, and it’s also very helpful that MMM is a small-data problem, so you don’t have to pull in terabytes of data.
With Robyn, especially, you don’t need much more than the ability to write natural language. You might not even need a pro or plus subscription.
The word “democratization” gets tossed around a lot. What’s truly getting unlocked for marketers here?
People can just ask the LLM to explain everything and refine questions for them to bring to an actual data scientist. The risk, though, is overconfidence. If you dive into the advanced modules without being able to vet what’s happening, you can end up making bad decisions based on analyses you hallucinated together with the LLM.
That begs the question: Can you actually trust the output?
You can trust LLMs to execute what you prompt them with, but you still need to supervise. And I’d push back a little on the framing of hallucinations as purely bad. People who have new, breakthrough ideas are, in a way, seeing things that the current consensus on knowledge hasn’t validated yet.
You can arguably use an agent’s hallucinations to see things that you didn’t realize before – but you also have to be the one to judge whether it’s promising or can work. That’s where the complementarity between human and machine intelligence really matters.
Where does measurement go from here? What does a measurement stack look like in, say, three years?
I recently asked on LinkedIn whether the confluence of open-source MMM and agentic AI has solved marketing measurement. I was being purposely provocative, and a lot of people pushed back. But I think there’s something to it.
However, I’d expect a trifecta of MMM, experiments and attribution to remain the state of the art for the foreseeable future – what Meta calls the “suite of truth.” MMM gives you the strategic view, experiments provide the ground truth to calibrate against and attribution is more about the tactical day-to-day.
MTA still has a place, but it needs to be scrutinized. I’ve heard from practitioners that different models can diverge by several orders of magnitude, and that’s something you really want to cross-check through experiments.
This interview has been lightly edited and condensed.
For more articles featuring Julian Runge, click here.
