AUDIENCExSCIENCE is where the industry comes together to present the most timely and exciting measurement research. See the 2021 highlights:
Scott McDonald, Ph.D. — CEO and President, ARF
The 80/20 Challenge: Building a Better Measurement Blueprint
Kelly Abcarian — EVP, Measurement & Impact, NBCU
NBCU’s Kelly Abcarian marshalled the call for revolutionizing opportunities in the measurement space and challenged attendees to drive change by questioning how they consume content, how they connect with that content, and ultimately, why, with current technology’s available options, is measurement still the same.
Kelly named this call to action “measurement independence,” a freedom from legacy thinking to seek the best solution for consumers and advertisers. Rather than being led by opportunity, she charged marketers to lead the way themselves.
Acknowledging that change cannot happen overnight, Kelly concentrated on how marketers currently focus 80% of their time on system failures and 20% looking for new solutions, asking “what if” this mindset was flipped and 80% of that energy was poured into innovative action instead.
Kelly then identified three specific challenges where measurement partners provided opportunities for custom solutions. She noted that it was the evaluation of these problems that would help marketers find the right partners for their individual needs and collectively create a pathway to a new measurement approach.
- The mismatch between old measurement methodologies and new technology: Assessing Conviva’s strengths as a measurement innovator, Kelly noted the company not only integrated data but also improved the experience by keeping pace with the consumer.
- Capturing consumer emotions: The ability to correlate emotions with creative and ad performance brought Dumbstruck to Kelly’s appraisal as a neuro-research company that captures facial expressions and eye movements to measure viewers’ conscious and unconscious feelings about the content watched in real time.
- Data accuracy: Where marketers are struggling to demonstrate ROI because most third-party consumer data is up to 50% inaccurate, Kelly distinguished the start-up, Truthset, as a possible partner who measures the accuracy of demographic consumer data with scores and independent datasets.
Kelly emphasized that these consumer-centered companies were just three of the potential opportunities for marketers and advertisers to consider on their way to measurement independence.
The Future of Audience in Streaming
James Lamberti — CMO, Conviva
James Lamberti, CMO of Conviva, a company that specializes in streaming and social media measurement and analytics, focused his presentation on marketers’ growing need for measurement of streaming video content, especially of viewing advertising on ad supported streaming services. In his view, measurement should address three key issues:
1. Accuracy: While he emphasized that measuring streams is “not easy,” James described his company’s methods which he said provide accurate measurement.
2. Interoperability for publishers: To achieve this important goal, James proposes standardized data sharing and a consistent data model across publisher’s brands and distribution points.
3. Viewer experience: The quality of viewers’ experiences is a major determinant of the ads’ impact and success. Therefore, we also should measure the technical quality of the ad exposure as well as making sure that privacy concerns are addressed.
- As streaming of ad supported video content is growing rapidly, marketers need better measurement of consumers’ exposure to advertising on streaming services.
- James Lamberti argued that measurement of advertising on streaming services requires new solutions. His company is focused on accuracy, standardization of measurement and the quality of viewers’ ad experiences.
- In sum, James Lamberti suggests that measurement of streaming should be based on census-level data that are publisher-controlled, single source, standardized and interoperable.
The Future of Media
Bill Livek — CEO & Executive Vice Chairman, Comscore, Inc.
Scott McDonald, Ph.D. — CEO and President, ARF
The last year-and-a-half has presented the media industry with a number of unique challenges when it comes to measuring how content is being consumed across platforms. In this one-on-one conversation, Comscore’s Bill Livek joins the ARF’s Scott McDonald to discuss how today’s viewership behaviors have signaled the need for a more modern cross-screen measurement approach and outline the fundamental changes necessary to effectively transact on media today and into the future. The following are edited highlights from their conversation:
Scott: From the standpoint of a measurement company trying to get as granular as possible but still aggregating things up for the advertisers who find this difficult and complex to buy – how do you think about this in 2021?
Bill: This really is THE question. There’s premium video and video user-generated content (UGC). But UGC should not even be remotely close to the premium value to an advertiser.
Scott: At this conference the one category of papers that we saw that had a huge increase was around the direct measurement of attention.
Bill: We need engagement impressions of all types; we need to define what their content is (not just with length).
Bill: Branded advertisers look at those characteristics and they have to filter into media buying, as they are already in planning. We see the great agencies that are hired by the premium brands do a remarkable job of planning of these other data sets. But when it’s distilled into a buying metric of age and gender, I am not sure if we are doing justice to premium content.
Scott: Limitations of panels – what’s the workaround if we don’t have panels, what’s the alternative way of reconciling?
Bill: In my career we’ve seen response rates deteriorate … response rates in radio and TV (decades ago) were in the 60’s or 70’s, which essentially means that we are guessing. We have been guessing in what the panel does not measure…. Panels are interesting to collect other information that are not in big data sets. And leave it there.
Scott: Parting words of advice?
Bill: Have an open mind as we look at the future of measurement… We have an opportunity now to fix all the different verticals and how they intersect on how the consumer is being entertained and informed. And let the advertiser have those tools and let’s empower the media companies so that they can price appropriately.
Optimize Early & Often
Ken Archer — SVP, Product, Upwave
The presenter reviewed the current challenges of optimizing and reporting brand campaigns mid-flight. Specifically, stat sig was never intended for midflight optimization and reporting, and is misapplied, causing errors. The question marketers ask mid-flight is “how much confidence can I have that a tactic is helping the campaign, given the other campaign tactics”? Stat sig cannot accurately answer this question.
- Optimization metrics developed for mid-flight would empower better brand campaigns by shifting budget to the highest-performing tactics, modifying DSP bid factors early and often, encouraging negotiations with media partners, setting creative rotation weights, and optimizing publisher inventory.
- Mid-flight optimization metrics are intended to be checked, acted upon frequently, and stabilize fairly quickly. Only use stat sig & p-values after a campaign ends.
- Better mid-flight reporting enables marketers at mid-campaign to: see which tactics outperform the campaign and shift budget accordingly; monitor daily for changes; have a precise read on the exact amount of lift driven by each tactic at the end of the campaign.
Driving Discovery: How New Habits Are Shifting Shopping Behaviors
Ashfia Rahman — eCommerce Consumer Research Lead, Facebook
To evaluate how people’s behaviors have changed, Facebook commissioned GfK for an online study of 1,002 U.S. adults 18+ in Q3 2020. The results confirmed that shoppers, slow to adapt pre-pandemic, turned to ecommerce in a shift that happened in two months and accelerated the overall online shopping mindset to “always shopping.” Facebook differentiated this movement from e-commerce (where people find products) as “discovery commerce”, where products find people.
- Facebook found that with the move to digital, most product discovery is happening online: 84% of surveyed online shoppers discover new brands and products online; 54% of those who discovered online saw the brand/product while browsing online and wanted to buy it.
- As this shift becomes more commonplace, shoppers’ approach to shopping is becoming more spontaneous: 65% of online shoppers say the internet is making them more spontaneous or impulsive in their shopping over time.
- Among online shoppers who discovered a new product online, nearly three out of four spend a day or less on decision-making: 35% spent 30 minutes or less; 73% spent a day or less; 86% spent a week or less.
- Facebook observed three common factors driving discovery commerce: relevance, endorsement, and ease.
- Relevance: Of the 91% online shoppers surveyed who use the internet for shopping inspiration, 63% were interested in the product they discovered because the options shown matched their needs and 36% purchased the product because it matched their interests.
- Endorsement: Online shoppers lean on advice from a variety of community advisors to fuel discovery and purchase.
- Ease: Reducing friction can help people move from discovery to purchase. More than one in three online shoppers surveyed want improved comparability on brand/retailer websites, and more than one in four want better ability to imagine products on brand/retailer websites.
Assessing the Potential of Addressable Linear TV Advertising
Rex Du — Professor of Marketing, McCombs School of Business, University of Texas at Austin
Susmita Ghose — Head of Data Science, LG Ads
Tsung You Hsieh — Doctoral Student, Bauer College of Business, University of Houston
Traditional linear TV places the same ads in the same shows (“program targeting”). Addressable TV can place different ads in the same show, allowing “audience-based” targeting at the household (HH) level. To determine the incremental lift achieved by audience-based targeting, we need to measure outcomes at the individual (HH) level. This allows us to run what/if scenarios with the models to determine optimal targeting. The researchers analyzed second-by-second viewing data from a panel with 750,000 HHs, tracking exposure to focal and competitor ads. The fact that advertisers tend to mainly target consumers already likely to buy creates potential endogeneity. The analyses controlled for this and three other factors: heterogeneity in ad avoidance; activity bias; and seasonality and other trends. During the observed 15-month period, there was a 4.1% conversion for those HHs in the market for the focal offer.
- Compared to the benchmark current targeting strategy, what/if analyses show that optimizing program targeting can increase lift by 90%.
- Optimizing audience targeting can increase lift 243%.
- When the cost of targeting is less than 1.3 times that of the current strategy, the targeted strategy will be cost efficient.
Capitalizing on the CTV Opportunity
Eric Cavanaugh — SVP, Cross-Platform Research, Data Intelligence, Publicis Media
Cara Pantano — Senior Manager, Sales Insights, Verizon Media
The CTV landscape is experiencing a plethora of new streaming entries, leaving consumers overwhelmed. This includes options that provide a tidal wave of content, some of it at lower cost. Publicis and Verizon partnered to explore how consumers’ expectations of services, content, and ad exposure will evolve.
The CTV experience is fragmented and nuanced, with multiple tiers available. Each has a role to play:
- Cable: Delivers on sports, live events, and easy-channel surfing. Also provides a “lean-back” escape and passive viewing.
- SVOD: Excels for varied, original, and premium content that are “must see” shows.
- Free AVOD: Nothing to lose. Driven by free-trials, consumers gain access to content with low-financial risk.
Choosing to watch SVOD over AVOD is not a rejection of advertising. Half of all CTV is ad-supported, driven by YouTube and Hulu. Ad experiences deemed “The Worst” include encounters with too many breaks, repetitive ads, and ads appearing at inopportune moments. Ad experiences that qualify as “Better Experiences” include approaches that give viewers more control, countdown clocks, and ads that are entertaining or personally relevant.
- The CTV landscape is experiencing countless new streaming entries. Consumers are overwhelmed and in search of options that gives access to premium content at a lower cost.
- Each platform plays a different role in the viewing ecosystem and can complement each other.
- There is an opportunity for AVOD to evolve to its potential at that moment of expanded awareness and interest.
The Evolving TV Streamer
James Lamberti — CMO, Conviva
Steven Millman — SVP, Global Research & Operations, Dynata
How do viewers feel about advertising on streaming services and how can it be improved? Research by Conviva and Dynata addressed these questions which are becoming more and more important to the industry given the strong increases in viewing of ad-supported streaming services. The study data were obtained through a February 2021 survey by Dynata and Conviva’s proprietary Stream Sensor™ technology. The analysis focused on identifying the key drivers of viewer satisfaction.
- Most viewers say they are happy to watch ads in return for free streaming services, but they nevertheless skip ads when they can.
- There are distinct segments with different views and behavior regarding streaming ads. In general, viewers appreciate lower ad loads and ads they regard as relevant and that fit well into the context. Technical difficulties, repeated ads, and privacy concerns are key drivers of dissatisfaction.
- The authors think that there are “Untapped Potentials” – viewers who would increase their viewing of ads (as well as of the content) if the ad experience was improved.
Cross-Platform: Measurement & Identity
Australia’s Largest Cross-Platform Study: The Power of News
Steve Bellman — MediaScience Research Professor, Ehrenberg-Bass Institute, University of South Australia
Duane Varan — CEO, MediaScience
Steve Weaver — Head of Research, Premium Content Alliance
According to the study’s sponsor, the research was prompted by many advertisers’ “ambivalence” about news as context for their ads. The study, the largest of its kind ever conducted, used an experimental design, placing comparable ads in newspapers and on digital platforms. Digital ads were placed in news and other contexts. The study was conducted in Australia, but U.S. data were also obtained. The analysis focused on three issues:
1. Does the news context promote ad recall, positive attitudes about the product, and brand choice?
2. Do ads work better on print or on digital platforms?
3. What do the findings tell us about how advertising works in different contexts and on different platforms?
- The study found significantly higher recall for digital ads in a news environment than in other contexts. Among light users of the advertised product category, news context also outperformed non-news in brand choice.
- Ads in newspapers were more effective than those on digital platforms in a news context.
- The researchers conclude that “Context Matters” as ads in news environments, both print and digital, benefit from the users’ cognitive involvement with the content. At the same time, the findings suggest that “not all news is equal”: it is important that users find the content relevant and trust the source.
Synthetic Solution: Leveraging Data Dependencies in Cross-Platform Measurement Models
Sean Pinkney — Sr. Director, Data Science, Comscore
In this presentation, Pinkney illustrated several examples of ways in which missing data can be imputed when measuring audiences across platforms through analysis of “data dependencies.” Those dependencies in one known set of data can be leveraged through a model to generate estimates of unknown data.
- Scenario #1: If there are data on site visitation rates in a desktop panel and data on site visitation rates from a mobile panel, how does an analyst determine the reach of the site on both platforms? If an analyst has data on both platforms from another smaller sample (which can be as small as 200 people), she or he can estimate the joint visitation rate of the site on both platforms. This can be done even if the desktop site visit rate in the small panel doesn’t match the site visit rate in the desktop panel and/or the same is true of the mobile site visit rate. Pinkney would assume that the odds ratio (the ratio of the product of the diagonals in a 2 x 2 table with one dimension representing did/not visit the mobile site and the other dimension representing did/did not visit the desktop site) is the same in the small sample as in each of the larger platform-specific panels. The estimate of the joint visitation rate then has to be calculated through rim weighting so that the resulting marginal visit rates in the panels are maintained.
- Scenario B: If there is a correlation matrix with one missing correlation, can the remaining correlations – the other dependencies in the data set – be used to help estimate the missing correlation? In the example shown by Pinkney, the missing correlation coefficient would be constrained to fall within the boundary of -0.87 to -0.40, given the other coefficients in the matrix. (The methodology for estimating those boundary points may be found here.) Pinkney estimated the missing coefficient to be -.64, the mid-point of the range. It turned out to be off by only 0.06.
- Scenario C: Creating segments by looking at searches and sites visited by people in both desktop and mobile panels. In this case, Pinkney deployed machine learning (ML) to try to discover non-linear relationships between search terms. For example, searches for Starbucks might be related to searches for coffee or Dunkin. Searches for Blizzard turn out to be related to searches for video games, because Blizzard is a company that makes video games. Pet intenders could be identified by selecting panelists who search for terms like adoptapet.com and looking at the other sites they visited during the same session.
- Scenario D: Pinkney’s fourth example was about “matching” members of a mobile panel to individual members of a desktop panel through machine learning that leverages data dependencies and marginal visit rates as in Scenario A.
- Scenario E: Longitudinal data over time can also be leveraged to discover dependencies which can help in creating synthetic estimates of missing data for entire countries.
Attribution & Approaches
Fast(er) Causal Attribution
Lindsey Woodland — VP, Analytics, 605
Alex Freed — VP, Account Management, 605
Michelle Gaudet — Group Media Director, Media Hub
Sunil Soman — VP, Marketing Sciences, WarnerMedia
The presenters analyzed the past attribution challenges for Chipotle and proposed new measurement solutions. Chipotle wanted proof that its TV commercials drove sales. Chipotle and WarnerMedia developed an outcome-guaranteed deal based on sales lift, rather than impressions. For Chipotle, incremental transactions are most important. These transactions were also measured for the competitors. The goal was to provide purposeful, visible, and accountable results.
Using these new solutions, Chipotle generated $4.57 in sales for every dollar spent, including $2.30 of incremental revenue. Future shopper target segments and media recommendations were also developed based on the refined data and methodologies.
- The new measurement paradigm provided expanded sales data (the pandemic stimulated third-party sales), greater granularity of competitive audiences, data collected in real time and analyzed more rapidly, impression-based calculations (MTA) rather than HH-based calculations, as well as additional media granularity measurement.
- Large-scale, high-quality viewership data for cross-platform measurement, analytics, and attribution addressed the biggest challenges facing the marketplace. Rich viewership data enabled powerful measurement for Chipotle.
- Improved cross-platform campaign measurement was achieved by using a matched control study. The lift estimates became more refined and reflected the true causal impact of the ad.
- Expanded measurement data of sales, audiences, and media can increase sales and incremental revenue outcomes, as well as provide direction for target segments and media planning.
- Guaranteeing business outcomes provides greater accountability.
A New Metric for Brand Loyalty
Leslie Wood — Chief Research Officer, NCSolutions
Amy Crooks — Senior Manager, Research & Development, NCSolutions
There have been dramatic changes in loyalty due to the pandemic, requiring a re-examination of the measures of loyalty, churn, and the value of new vs. loyal buyers. NCSolutions analyzed why it is important to compare loyalty measures longitudinally to gauge a brand’s health, as well as to understand which advertising and promotional strategies have been successful and to determine whether to focus on driving penetration or brand loyalty.
The presenters examined case studies for a baby food brand and a facial cleanser brand with the key metric of churn; a salty snack brand and a beer brand with the metric being depth of repeat purchases; and a frozen food brand with the key metric of YOY brand $/HH. The case studies examined different buyer cells for responses to the creative over time.
- The depth of repeat purchases (the sum of incremented, consecutive brand purchases) is central to the brand’s long-term health and a key measure used in measuring long-term effects of advertising. The higher the depth of repeat, the stronger the long-term effects and the higher the buying the following year. This measurement allows a household to be classified as loyal to more than one brand in the category.
- Brand loyalty metrics are very powerful and provide a dynamic view of the brand over time. The essential metrics to understand a brand’s buyer dynamics are churn, average depth of repeat, and YOY brand $/HH difference. Different scenarios call for the use of different metrics.
- The NCSolutions’ analyses and resulting insights enabled effective consumer targeting and adverting strategies to be developed. Great creative can drive customer retention and acquisition.
Modelling Short & Long Term Marketing Effects in the Consumer Purchase Journey
Dr. Peter Cain — Executive Partner, Marketscience
This presentation is based on the latest IJRM publication focusing on short and long-term effects in the consumer purchase journey, focused on two key issues facing marketing response modelling. Peter explained that the standard marketing mix model (MMM) fails to resolve two key issues: it ignores inherent selection bias of online media and the true mechanics of brand-building. Both issues result in significant mis-estimation of ROI and incorrect marketing resource allocation.
He proposes that a more appropriate econometric estimation approach to MMM combines both time series modeling of short-term demand (through standard marketing metrics like paid media, search, clicks, and short-term sales) and long-term modeling of base sales, customer journey, and brand metrics. By incorporating long-term base sales and branding effects, these models provide results that more accurately reflect the context within which short-term marketing effects occur.
Peter shared that he started as an academic economist before transitioning into the marketing analytics industry. His goal is to provide transparent, analytical approaches for his clients. One way to achieve this is by publishing their approaches.
- Traditional MMM measurement approaches do not provide a robust understanding of the true impact of marketing activity.
- Ignore selection bias of last-touch online media
- Short-term focused by construction
- Standard long-term measurement techniques are flawed and ignore time-series properties of the data
- A preferable approach combines:
- Dynamic time series models to capture short-term effectiveness and extract long-term base sales
- A network model for brand-building, capturing interactions of paid media, base sales, brand metrics and earned (social) media
- Long-term preferences embodied in attitudinal data causally linked to long-term preferences reflected in base sales
- A credible structural framework for quantifying the emotional foundations of brand-building media campaigns
Attention Measures: What Counts & How Much Does it Cost?
Understanding the True Cost of Attention Across Media
Mike Follett — Managing Director, Lumen Research
Yan Liu — CEO & Co-Founder, TVision
Lumen Research and TVision came together with Ebiquity to study differences in the way advertising generates visual attention across varied media and how much it costs to buy that attention. By combining their individual datasets – Lumen’s visual attention to digital advertising on desktop and smartphones, TVision’s TV attention data, and Ebiquity’s cost data – the researchers devised a new currency, the aCPM, a proxy metric representing the cost per thousand seconds of attention.
Suggesting that this attention CPM is a key lever for ROI enhancement, the study results showed that attention to TV and YouTube consistently outperformed social and desktop platforms in terms of cost-effectiveness.
- The longer consumers look at ads, the more likely they are to work.
- Ads that generate more attention are also the ads that generate the most sales and ad recall.
- There are big differences in the amount of attention that goes to different forms of advertising. Buying the best media is very important- but generating attention from the best creative is equally important.
Does Every Second Count?
Kara Manatt — SVP, Intelligence Solutions & Strategy, Magna Global
Heather O’Shea — Marketing Science Lead, Snap, Inc.
Kara Manatt (Magna) and Heather O’Shea (Snap) presented research that compared :06 second and :15 second ad lengths across three video platforms – Snap, video aggregators, and full episode players (FEPs) – to determine the optimum ad length for an effective ad strategy.
In testing the same :06 and :15 ads for the same four brands, the study factored in the characteristics of each platform – pre-roll/mid-roll, skippable and non-skippable, and device – as it tracked 7,500+ panelists’ viewing behaviors for brand awareness, brand perception, and purchase intent.
- Video ad length is not a predictor for ad effectiveness. Both :06 and :15 ads can drive persuasion metrics. Planning should be based on more than ad length alone.
- Ad lengths are device agnostic. While ads generally perform best on mobile there’s no difference in performance of :06 and :15 second ads across devices, making planning ad lengths by device not necessarily recommended.
- Fit the ad length to the platform. Advertisers should consider matching the length of the ad with the platform and length of the video content being consumed.
- Shorter ads “win” on platforms with shorter content, like Snapchat and video aggregator platforms.
- Longer ads are a better fit in environments with longer form content, on platforms like FEPs with TV-length programming.
Advertising’s Sequence of Effects on Consumer Mindset and Sales
Shuba Srinivasan — Professor of Marketing, Boston University
The academic study at the heart of this presentation compared 13 hierarchy-of-effects (HoE) advertising models to determine which model matters the most, what moderators are most prominent, and what factors and sequence are most important in driving sales. Understanding the sequence of effects is most important for advertisers and marketers as they build their campaigns.
In examining 178 FMCG brands in 18 categories from Kantar’s Worldpanel brand performance data, the researchers found that the predominant form of hierarchy that mattered most was the integrated HoE that triggered the sequence of Affect (feeling/emotion) first, followed by Cognition (belief/knowledge of brand) and Experience (memories of purchase or consumption of brand), or ACE.
- Integrated HoE framework generalizes across brands: 94% of brands (168 of 178) have integrated HoE as operating framework; 4% of brands (6 of 178) have Classic HoE as operating framework.
- ACE (Affect/Cognition/Experience) is the most common sequence.
- ACE was even more prevalent for less differentiated brands, and for utilitarian (FMCG, detergent, etc.) vs. hedonic (beauty) products, which was less effective.
- Cognition is most responsive to advertising followed by Experience and Affect.
- Understanding the sequence of effects is most important. Last factor in sequence drives sales most at 55%, 27% from middle and 17% from first.
Session Chair: Helen Katz — EVP, Research, Publicis Media
Cross-Platform: Measurement & Identity
Session Chair: Cole Strain — Senior Director, Audience Products, Samba TV
During the discussion, Duane Varan (MediaScience) and Steve Bellman (MediaScience & Ehrenberg-Bass Institute) discussed the metrics they used, the implications of their research for media planning, and the results of their research in the U.S. Sean Pinkney (Comscore) talked about the lessons he had taken away from his analysis of missing data. Below are edited highlights from their discussions.
- The metrics Duane and Steve used correspond to three stages of memory:
- The retrieval stage: Obtained through an unprompted recall question like “Do you remember any brands you saw in this session?”
- The storage stage: Obtained through a cued recall requestion like “You may have seen some ads from brands in this category. Can you remember any brands advertised in this category?”
- The encoding stage: Obtained through a brand recognition question in which the brand and its competitors are shown, and the participant is asked “Which brands did you see ads for?” This can also be a proxy for attentiveness.
- Cole Strain (Samba TV), the moderator, said that he was impressed by the impact of digital news on ad effectiveness that Duane and Steve had found. He was even more impressed by the massive impact of print. Duane commented that we’re very alert to context in every non-digital environment; for example, we are aware of the way that an outdoor ad may have a different effect than an ad in a television sports program. But on digital, we treat all exposures as if they are the same and that context doesn’t matter. Their study shows that context does matter in digital. Steve added that “We’ve all forgotten how powerful newspapers are” and that advertisers should keep advertising in print editions as long as they can.
- Continuing his thoughts on context, Duane remarked that the absence of any lifts in the U.S. bore out the importance of context. Since the advertised brands don’t exist in the U.S. and since the news stories were all about Australia, it was not surprising that the appearance of an ad in a news context did not deliver the benefit that it did in Australia.
- Cole asked Sean about when his approaches to extrapolating to obtain estimates of missing data “go too far.” Sean replied that it can be hard to build trust in synthetic estimates, acknowledging that the word “synthetic” arouses suspicion and that “imputation is a tough sell.” He recommended the following steps:
- Understand how the data are created and how they can interact across space and time.
- Attempt to develop a “principled way” to estimate missing data, rather than just “mindlessly imputing data.”
- Place the assumptions behind the models “front and center.”
- Clearly communicate limitations of what you’ve done. In Sean’s words, a modeler should “quantify the uncertainty.”
- Show the level of confidence around the estimated values.
- Check how well the model performs with a hold-out sample of known data.
- Iterate to create better models.
Attribution & Approaches
Session Chair: Paul Donato — Chief Research Officer, ARF
In this discussion for the track, Attribution & Approaches, session chair, Paul Donato (ARF) asked the speakers for their key insights on the drivers of short-term and long-term sales, the role of match control, and whether testing control should be part of attribution and ROI. Below are edited highlights from the discussions.
- The drivers of short-term sales include price, promotions, calls-to action, performance TV, noted Dr. Peter Cain (Marketscience). Long-term effects are about brand building with emotional constructs. Promotions can drive short-term sales but denigrate brand equity.
- Paul stated that he often quotes Leslie Wood (NCSolutions), that it is not possible to have long term sales without short-term sales.
- Media measures, such as reach and GRP, matter short-term. Amy Crooks (NCSolutions) observed that the depth of repeat sales is the key to long term health of the brand. Leslie added that short-term and long-term are related. What happens short-term does impact long-term, but not all the pieces. Brand metrics are included in NCS’s short-term analysis. Loyalty is a key component of long-term.
- Peter added that it is hard to know what drives repeat sales. There are many factors other than media, such as consumer experience with the product, product performance, and the quality of the product, that need to be considered in terms of long-term sales.
- Leslie countered by pointing out that NCS did two studies looking at short-term effects and revisited those same households in terms of their buying behavior, in the following year. Using real data, NCSolutions demonstrated that the number of consecutive purchases of a brand in a category matters long term. It is a driver of incremental sales and increases category sales long term. Leslie agreed with Paul’s statement that a short sales cycle has a higher multiplier effect than a long sales cycle.
- On the subject of randomized control groups vs. matched control groups, Sunil Soman (WarnerMedia) noted that an in-depth understanding of incremental control and match control is needed. Looking at the macro level, randomized control groups will work; however, matched control groups are essential for a more details analysis. Matched control allows for the impact of ads on exposed audiences. Lindsey Woodland (605) added that in the absence of the gold standard of random control trials with sufficient sampling, matching modelling controls for outside factors. A synthetic or modeled approach is not recommended.
- Paul commented that an MRC standard is being developed for attribution and ROI. Testing control was removed from those standards as it gave the advantage to walled gardens. Paul asked the panel if testing control should be part of attribution and ROI. Sunil and Lindsey agreed that matched control should be part of the standard. Matched control methods show causality better.
- Amy stated that NCSolutions uses statistical methodology to build models. Random control should not be part of the standards. Testing control provides similar results to the NCSolutions model. However, Leslie cautioned against randomized control and pointed out that testing control is very broad in definition.
- Peter noted that a regression context is needed. He is concerned about selection biases. Standard regression tells researchers nothing about causality.
Attention Measures: What Counts & How Much Does it Cost?
Session Chair: Jane Clarke — CEO, Managing Director, CIMM
Jane Clarke (CIMM) followed up with each of this session’s presenters on the goals and data points of their discrete studies. The following are edited highlights from the discussions.
- A necessity condition is that consumers have to pay attention to advertising for advertising to initiate any kind of sequence, according to Shuba (Boston University). To the extent that consumers pay attention to ads, only then is any kind of advertising effect through a hierarchical sequence triggered, so it’s a necessary condition but it’s not sufficient to say which of these intermediate factors would have the effect on sales. Not all of these metrics drive sales equally – know the sequence for your brand and advertisers.
- Gen Z and Millennials consumed more content overall, but still had a higher rate of aided recall than other generations (Gen X, Boomers), shared Heather (Snap). Last year, they conducted a research study with Kantar to evaluate the information processing power across different generations to see if there were any differences. Each generation used Snap as they normally would, and they controlled for ad exposure. What they learned is that younger participants showed superior ad processing power when looking at ad message recall. This is surprising because we may be underestimating what we expect from the younger generations.
- Advertisers are getting better at creating 6-second ads. According to Kara (Magna Global), back when they first started building :06 second ads, it was simply taking your :15 or :30 second ad and cutting it down to :06 seconds. You were really at the mercy at what had already been shot for another purpose. Cutting the original down to :06 seconds and maintaining branding and storytelling was very difficult to do. Now advertisers are creating :06 second ads – either on a custom basis or shooting with :06 second ad in mind, knowing that the longer versions will be cut down. Overall, that’s led to more efficient short ads because they’ve learned with the right material and testing what is going to work in a shorter amount of time.
- The historical econometric model approach won’t garner the most accurate view of cross-platform reach or delivery, noted Heather. From this research they were able to provide a different way of thinking. A :06 second ad isn’t half as effective as a :12 second ad, and a :12 second ad isn’t a frequency of 2 to a :06 sec ad – that kind of thinking doesn’t hold true any longer. They saw that there were other kinds of descriptors, like platform, device, attention – those can and should be used to better equivalize impressions across platforms. She hopes this research challenges the industry’s way of thinking.
- A new tool called the Attention Calculator was just launched by TVision and Lumen. Yan (TVision) explained that this tool was based on their study and it’s for anyone interested in attention for media planning and duration based metrics. It’s a free and interactive tool that calculates the cost of attention with the user’s CPMs to see the average cost per impression across platforms, based on Ebiquity data.
Scoring and Representation
Radha Subramanyam, Ph.D. — Chief Research & Analytics Officer, CBS Corporation; President, CBS Vision
Scott McDonald, Ph.D. — CEO and President, ARF
Scott welcomed Radha to their virtual fireside chat on the subject of representation in media and the workplace. Scott recalled that at ARF AUDIENCExSCIENCE Conference in 2020, CBS outlined its steps to improve diversity and inclusiveness as both a moral and a business imperative. He asked Radha about CBS’s current thinking.
Led by Radha, herself a pioneer in the media and technology industry, CBS Research, CBS Diversity and Inclusion, and CBS Entertainment are working to explore a more inclusive future for all media. CBS has been at the forefront of programming inclusiveness since they started using the Gender Equality Measures (GEM) for their program research in 2018. In 2020, CBS took the next step and adapted the GEMs, which score consumer reaction to the treatment of women in advertising programming, to form a BIPOC metric assessing the value of racial representation in people’s content choices.
- The business case for DEI is simple and obvious, according to Radha. The U.S. Census revealed 57%/43% white/nonwhite population statistics. CBS cannot exclude 50% of its audience. It is essential to serve your whole customer base.
- To tell authentic stories about a diverse audience, behind the camera is just as important as in front of the camera. Rada’s boss, George Cheeks, went public with the goal of 40% of writers BIPOC. CBS exceeded this goal and upped the target. Additionally, the BIPOC unscripted goal of 50% was exceeded. The response from employees has been overwhelmingly positive, as shown in both hard metrics and anecdotes.
- CBS’s own diversity metrics were developed inhouse and go beyond GEM and BIPOC metrics. These metrics were tested on two pilot seasons of shows and promotions. An emphasis on diverse programming resulted in programs such as United States of Al, The Equalizer, and Your Honor.
- Are the industry’s current forms of measurement adequate to measure diversity? Only the surface has been scratched. There is a need to make changes in our measurements. Panels do not reflect the diversity of the U.S., such as the Latinx and Asian communities. Truthset shows that African American targeting is only 27% accurate. The industry can also do better on gender accuracy.
Cross Channel Measurement in a Time of Data Collection Challenges
Jennifer Pelino — EVP Omni Channel Media, IRI
The average home has over 300,000 items, and consumers may be exposed to 6-10,000 ads daily. We need to overcome measurement silos to truly understand what triggers the different paths to purchase for the same products. Third party data sources need to be vetted on their sources and collection techniques, their validation methods and how they help us understand traditional metrics such as recency, frequency and consistency. Loyalty card data can help CPG companies track the 90% of purchases that still occur offline. IRI’s retailer and other partnerships offer a more holistic view of purchase behavior. In a masked case study, COVID reduced linear and cable but increased connected TV viewing, putting a premium on “equity spots” for in-home occasions such as food preparation/consumption. Multi-touchpoint fractional attribution (MTA) can distinguish the impact of creative from other aspects of digital ads.
- For the emerging new advertising ecosystem, we need a more holistic and transparent view of the entire purchase journey.
- A key requirement is linking identities across multiple devices, leading to higher-quality data and consistent cross-channel targeting.
Welcome to the Attention Dimension: Fundamentals of Attention Metrics for Media
Marc Guldimann — Co-Founder and CEO of Adelaide; Co-Founder, The Attention Council
The speaker, Marc Guldimann, founder of Parsec Media and co-founder Adelaide, is also a co-founder of The Attention Council. In his presentation he explained why he and the Council are promoting the use of attention metrics to help advertisers place ads into media environments that drive superior performance.
Ad impact is affected by media, creative, and relevance to the consumer. To optimize media, Marc argued, traditional measures have shortcomings: viewing duration, for example, does not have a linear relationship with outcomes. (Research shows the first seconds are usually more important.) He suggests that advertisers should optimize media by focusing on media quality and measure which media contexts help ads to get noticed and hold attention.
Marc described how his company develops models that quantify various contexts, generating data-driven insights around the quality of media that allow marketers to experiment and develop a media placement quality score that predicts business outcomes. (Note: Other attention measurement companies focus on optimizing creative with attention measurement.)
During the last couple of years, interest in attention measurement has grown rapidly and some advertisers have built attention guarantees into contracts.
- The Attention Council is promoting the development of attention measurement and the use of attention metrics by advertisers to optimize ad placement.
- The presenter argues that attention measures are superior to metrics such as duration and viewability based on many studies showing correlations between attention metrics and ad impact measures.
- Testing and the developments of models is recommended to optimize media spend towards placements in those media environments that show higher potential for attention.
THE LAST WORD
Megan Margraff — VP, Data Strategy, Oracle Advertising
Jennifer Pelino — EVP Omni Channel Media, IRI
Radha Subramanyam, Ph.D. — Chief Research and Analytics Officer, CBS Corporation; President, CBS Vision
This panel discussion closed out the first day of the conference, highlighting the anchor commentators’ key takeaways from the day’s sessions. The following are edited highlights from the conversation:
- The gaps highlighted in today’s presentations really show an opportunity for the industry to close those gaps and continue to evolve, according to Megan (Oracle). For instance, in the morning session lead by James Lamberti (Conviva), he mentioned a gap between buyers and sellers in the confidence to buy OTT inventory. This illustrates an opportunity for a common currency to transact on media in a meaningful way while dealing with a very fragmented ecosystem and with the fact that not all impressions are created equal.
- Jennifer (IRI) concurred and expressed surprise at how big the gap was between buyers and sellers on what was deemed at having the right information. All of our jobs now in the industry involve helping minimize those gaps. Additional challenges involve duration, placement, targeting, and managing frequency.
- Overall, there was a high degree of consensus both in terms of the gaps and the opportunities, according to Radha (CBS). There’s an agreed understanding that not all attention, not all impressions, not all contexts are the same. We have all collectively thrown off the “measurement shackles of the past,” and are embracing new approaches, including a multi-currency future with a suite of solutions.
- Consumer-centricity in privacy was also a common theme. Megan highlighted the big gap between consumer perceptions of privacy protection and buyers/sellers’ views on privacy compliance, and observed that having consumer trust and making any solution consumer-centric is really critical to our success in the industry. Jennifer stated when we think about collection of data, especially for consumer products, we have to think about the way that the consumers will provide that data and think about the value exchange (e.g., loyalty card data is the “purest form of transaction of privacy”). Radha also noted the dynamic of wanting to be more open – across partners and different types of data.
Scott McDonald, Ph.D. — CEO and President, ARF