Analytic Partners’ Trent Huxley interviewed client Mallory Fetters of Cox Communications on the telecom’s marketing measurement strategy. Dealing with constantly evolving challenges, both common and specific to its industry, Mallory expanded on how data deprecation, shifting consumer media behaviors, demand for faster speeds and growing consumer choice and competition rapidly accelerated Cox’s learning curve.
Warner Brothers Discovery (WBD) worked with Civis Analytics (CA) to optimize their TV programming promotions over 30 U.S. networks that premieres dozens of seasons annually across a diverse linear TV portfolio. Max Schuman explained how CA’s approach blended classic marketing mix model (MMM)’s regression models with machine learning’s (ML) ability to discern relationships that best predict outcomes humans can’t see easily. With a custom model that was able to guide decision-making on several levels—what TV series to promote, how and where to market, and what ROI to expect—WBD used CA’s platform as a starting point for all media decisions throughout the full funnel, inclusive of owned and paid media.
Greg Dolan (Keen Decision Systems) and Mark Bennet (Johnsonville Milwaukee) examined how to navigate in uncertain and volatile times in the current marketplace using next-generation marketing mix solutions. In the opening, Greg explored the progression of marketing from the late 90s, through what he dubbed “The Roaring 20s.” He noted that we went from a minimally complex, slower-paced “top-down” approach in the 1990s to a very fast-paced, complex environment with a shift to Retail Media and a unified approach where we apply next-generation predictive analytics in the 2020s. Greg discussed the intricacies of their approach of combining historical data with predictive/prescriptive plans to address drastic changes in the current environment, leveraging ML. He provided a case study that demonstrated the successful application of the next-generation marketing mix. In addition, Mark gave a client perspective on how they are handling market uncertainty.
Sameer Kothari (PepsiCo) and Todd Kirk (Middlegame Marketing Sciences) examined the application of a transformed rendition of marketing mix modeling created through the development of a proprietary system called the “ROI Engine.” Sameer indicated the desire to harness the “true potential of marketing mix models even beyond measuring past campaigns and using it for strategic planning looking forward.” Sameer discussed this system as having a more “predictive ROI outcome-based approach” by “leveraging an ecosystem of leading indicators for before and during a campaign flight.”
Liz Riley (OLLY) and Mark Garratt (In4mation Insights) explored rebuilding and reimagining marketing mixed modeling (MMM) to better handle fragmented data, in the era of retail media networks. Mark lauded MMM as an effective technique that has contributed to financial success for many businesses. In light of data becoming increasingly fragmented, he suggested that “some reinvention of the fundamental model framework is going to be required in order to move this old venerable method into the future.” Mark and Liz examined the Bayesian approach to MMM in handling fragmented data. Mark noted that there will not be a situation “where all the data is the same granularity in one place at one time.” The Bayesian approach can “fill in the blanks” of missing or fragmented data using reasonable estimates, creating a more accurate picture, which traditional MMM falls short of in the retail media environment.
Peter Cain of Marketscience covered shortcomings of marketing mix modeling (MMM). It is hampered by the endogeneity problem (selection bias) in things like online media. MMM is also only a short-term approach. The standard long-term method accompanying MMM is also flawed and ignores the time-series properties of the data.
In this session, Elea McDonnell Feit (Drexel University) led a panel discussion with the day’s speakers on innovations in experiments in marketing and referred to these experiments as a “mature part of the measurement system.” In this discussion panel members brought up ideas and examples of how to effectively employ randomized controlled trials (RCT) and the benefits of using experiments for attribution. They examined the lack of patterns stemming from advertising incrementality and credited this to the changing nature of the consumer journey and unique factors in strategy, the business life cycle and the product being sold. The panel also explored processes to ensure the deployment of a successful and effective experiment. In addition, geo-based tests were also considered. Other topics discussed were the cost-effectiveness of running experiments and the value of failed experiments.
The MRC’s Ron Pinelli outlined the scope of the Outcomes and Data Quality Standard, recently completed in September 2022. Part of MRC’s mission is setting standards for high quality media and advertising measurement, and Ron walked through the phased approach and iterative process that included the ANA, the 4A’s and other industry authorities.
Brendan Kroll of Nielsen and Anne Ori and Daniel Sacks, both of Google, explained that their study’s objective was to identify potential improvements to marketing mix models by utilizing enhanced prior beliefs (priors) based on sales lift studies and exploring the resulting changes in campaign-level sales lift once those priors were incorporated.
Daniel Sacks – Incrementality Lead, US Agency, Google
Brendan Kroll of Nielsen and Anne Ori and Daniel Sacks, both of Google, explained that their study’s objective was to identify potential improvements to marketing mix models by utilizing enhanced prior beliefs (priors) based on sales lift studies and exploring the resulting changes in campaign-level sales lift once those priors were incorporated.
Incrementality experiments are widely accepted as the gold standard for causal measurement. Calibrating individual channels via experimentation ensures optimization of model outcomes. However, the results of incrementality experiments are often not part of marketing mix model (MMM) design.
Nielsen utilized NCS sales lift studies as the source of the experimental data for this analysis. NCS determined the causal effects of advertising on incremental sales while controlling for targeting and other co-variates.
The study design involved 10 brands with existing MMMs and available NCS results for corresponding periods, model re-estimation using NCS lift priors, refinement of the priors and scaling. This study showed that applying this methodology to a YouTube campaign resulted in significant sales lift, as well as revenue and ROAS increases, including a 2.6x median increase in the effectiveness in the adjusted model.
The adjusted model showed greater marketing contribution overall; therefore, marketers are at risk of undervaluing their overall marketing if experimental results are not included.
Key Takeaways
Brands can effectively leverage experiment-based priors to strengthen marketing mix models.
For nascent channels, the inclusion of experimentation results proved fundamental, especially if those campaigns showed strong initial results, since MMMs cannot rely solely on historical anchoring to measure true impact.
When experiments reveal high performing channels or campaigns, the use of testing can aid more accurate MMM measurements as investment scales.
Even for channels with long histories and relative stability, experimentation can serve as a way to validate models and may give models a chance to remain flexible in case of strategic shifts and/or changes in consumer behavior.
