research quality

Making Sense of Multi-Currency Initiatives

Jon Watts (CIMM) led a conversation with the CEOs of an organization that is helping to manage the JIC (OpenAP) and one that participates in it (the VAB), the EVP of an organization that does not belong to the JIC but has met with it and the CEO of the MRC. The participants clarified their relationships with each other, discussed Nielsen and expressed their hope for the future of television measurement.

Use AI to Automate These Aspects of Market Research

  • MSI

Adopting AI usage into business functions seems to be the new trend, but can it be used to increase efficiency in market research? This study finds that it can help reduce costs and improve speed by automating some aspects of the process. One such way is to use Large Language Models (LLMs) as stand-ins for human survey respondents. The pool of respondents is shrinking. Moreover, the research finds that LLMs can realistically reflect consumer preferences because they have been trained on extensive online data. This process is also a fraction of the cost and time required with conventional methods.

Member Only Access

JIC: Coalescing Around Standards for Cross-Platform Currencies

Brittany Slattery (OpenAP), who opened this discussion, explained that the new JIC was created by national programmers and media agencies for three main purposes: (1) To bring buyers and sellers to the table with equal voices; (2) To create baseline requirements for cross-platform measurement solutions and (3) To create a harmonized census-level streaming service data set across all of the programmers in the JIC. Fox, NBCU, Paramount and Warner Brothers Discovery are all JIC members, as are Dentsu, Group M, IPG Mediabrands, OMG and Publicis. The members hope to foster competition among multiple ad video measurement currencies. After her introduction, Danielle DeLauro (VAB) moderated a discussion with the representatives of three networks and Group M.

Inclusion by Design in Pharma Research and Marketing

  • Pharma Council

This ARF Pharma Council event followed up on the Council’s podcast episode on “Inclusive Futures of Humancare,” focusing on the importance of inclusiveness in pharma research and marketing with respect to both demographic characteristics and health conditions.  Four speakers delivered brief presentations, followed by a discussion moderated by Pharma Council Co-Chair Marjorie Reedy of Merck.

Member Only Access

Let’s Face It: Facial Coding Isn’t Up to the Task

Elise Temple, Ph.D.VP, Neuroscience & Client Service, Nielsen IQ



There are limitations in terms of measuring the emotional impact of video. Facial expressions are powerful. Neuroscience has proved that special parts of our brain is dedicated to faces. Development psychology also demonstrates the power of facial expressions. Consumer neuroscience also demonstrates the importance of faces. The question is whether Facial Action Coding System (FACS) is the best tool to measure consumers emotional response when viewing ads? Nielsen IQ found that when you measure people’s facial expressions you need high quality videos, good lighting, algorithms. Nielsen IQ tests across over 2,000 ads in 15 countries. Initial R&D, global deployment, became a standardized diagnostic integrated with eye tracking. First phase found that facial expressions are only reproducible in strong and consistent moments. Happy is reproducible if it is really big, namely includes strong smiles. Negative emotions—reliability and reproducibility were as low as .25%. People watching TV, however, are mostly neutral. Reproducible expressions don’t occur often. Brain measure EEG, however, fluctuates much more and is highly dynamic. Put differently, there is a gap between facial expressions and EEG waves. Emotional expression does not equal emotion in the brain r=0.07. Is FACS predictive of anything meaningful? NO! No significant relation between ad-driven sales lift and any emotional expression. Facial coding is not the right tool for the job of emotional response to video ads. It is not reproducible—when measured continuously, don’t get same answer twice; it is not sensitive—at reproducible levels, expressions are rare; it is not meaningful—expressions do not equal emotion and are not reflective of dynamic emotion in the brain; and not predictive—smiles do not equal sales and don’t correlate with outcomes.

Key Takeaways

  • Facial coding is not a good measure of measuring emotional response to video ads.
  • Only smiles were found to register in somewhat reliable manner with facial coding.
  • There is no significant relation between ad-driven sales lift and any emotional expression.

Download Presentation

Member Only Access

Using Attention AI to Predict Real-World Outcomes

Max KalehoffVP Marketing Growth, Realeyes

Johanna WelchGlobal Mars Horizon Comms Lab Senior Manager, Mars



Mars and Realeyes prove connection between creative attention and sales performance. Mars’ Agile Creative Expertise (ACE) tool tracks visual attention and emotion responses to digital video ads. Visual attention AI and facial coding measures how long participants watch a video and how their attention changes as they watch. Proven this model to work—optimizing content, lifting sales up to 18% in 19 markets, $30 million in ad optimizations in 18 months. ACE can 1. Predict sales accurately while learning how consumers behave and think. 2. Optimize—improve performance through creative selection. 3. Scale—establish a fast scalable solution. The model links attention, emotion and memory. Accordingly: 1. Attention is access to the brain and enables the brand to enter into consciousness. 2. Facial reactions—build memories. 3. Impact—higher consideration, conversions and sales. ACE solution: 1. Participant exposure: 24-48 completion, 150-500 viewers from pool of +200m people. Observe people. 2. Attention detection: deep learning, collect viewer attention through natural viewer experience. 3. Actionable scores: ML and AI analytics to assess performance and deliver scores. Company impact: validated predictions proved connections to behavioral and sales data via over 4,000 sales/ads data points and benchmarks. They also used ACE to improve performances for TikTok, Facebook, Instagram, YouTube, achieving 18% cumulative sales lift. Global scale—scored over 1,000 creatives in 18 months. In conclusion, ACE is the biggest attention database, received U.S. patent for visual attention detection. Mars hopes to share ACE with other companies. And the next step is how to take a pre-testing tool to in-flight content and to examine brand equity.

Key Takeaways

  • Establishing the connection between creative attention and sales performance is key.
  • Mars’ Agile Creative Expertise (ACE) tool tracks visual attention as well as emotional responses to digital video ads.
  • Proven this model to work—optimizing content, lifting sales.

Download Presentation

Member Only Access

In-Home Psychophysiological Television Measurement

Pedro AlemidaCEO, MindProber

Lauren ZweiflerSVP Insights & Research, Client Strategy & Insights Org, NBCU



How to measure attention? This presentation introduced MindProber—a tool that includes biometrics, physiology, conscious/dial AND survey methods to assess attention. MindProber measurement is passive—second by second emotional engagement is measured through electrodermal activity (EDA), aka galvanic skin response (GSR) and also active—measures cognitive response via an optional feature to indicate like/dislike content through app. N= 1,500 and growing to 3,000 by end of year. Pedro Almeida (MindProber) emphasized three indicators of the value of metrics: 1. Validity—measures what it is supposed to measure. GSR is linearly related to perceived arousal in video formats. Measures emotional intensity (example of soccer game and spikes during key moments such as goals). 2. Reliability—metrics are stable across samples with 98.3% accuracy in ranking high vs. low impact ads. 3. Predictive power—metrics predict events of interest. Carry-over—if you’re more involved with the content you will respond more to ads. Premium content leads to higher advertising impact. Engagement with contents will carry over to engagement with ads. Emotional impact is indicator of likelihood of remembering the brand. Generating macro insights through a robust taxonomy—analyzed 150+ sessions, watched 250+ hours of programming, monitored 20,000+ individual hours, 10,000+ individual participations, 22,700+ events tagged and 8,500+ ads. Ninety-eight percent of commercial activity is as engaging as the content. Findings show 1. validity exists. 2. Reliability—98.3% accuracy in ranking high vs. low impact ads. 3. Predictive validity: A) Higher emotional impact for content = higher emotional impact for advertising. Carry-over—premium content leads to higher advertising impact; B) Higher emotional impact score = higher brand recall. Emotional impact is an indicator of likelihood of remembering the brand. Lauren Zweifler (NBCU) showed that MindProber successfully identified the most crucial moments on both scripted and unscripted shows (Iconic holiday TV moments for the first and Top Chef and America’s Next Top Model for the latter) WITH validity, reliability and predictability. What’s next?
  • How can we best predict lower funnel outcomes?
  • What elements of creative really matter?
  • How do we optimize for CTV?

Key Takeaways

  • New tool that captures emotional engagement through skin response.
  • Three fundamental metrics are validity, reliability and predictability.
  • Generated macro insights through a robust taxonomy. Power of premium video content yield strong emotional engagement (EE).

Download Presentation

Member Only Access

Why Visual Attention is a Flawed Measure of Attention

Duane Varan, Ph.D.CEO, MediaScience



Because of the complexity of attention, Duane Varan proposes to focus on inattention and regard attention as absence of inattention. Attention is the threshold above which the cognitive processing of stimuli occurs. It is not linear; it’s best understood as occurring above a threshold (inattention) after which other variables can deliver their value. There are many different kinds of attention, and the different measures capture the different types of attention. Inattention, on the other hand, is a constant construct. Phase 1 of the study focused on literature-proven measures (except fMRI) and found the following to be the four best methods: Heart rate (HR), visual fixations, blink duration, EEG (Alpha Wave). MediaScience and the Ehrenberg-Bass Institute concluded that eyes-on-screen is not best-in-class because 1. People are often paying attention even if not looking at screen; 2. Even when people look at the screen, they aren’t always paying attention, at least with fixations you see movements. Phase 2 used ads as stimuli and found two measures worked best: heart rate and EEG. Duane Varan pointed out that as an industry we have been fixating on eye tracking. But many ways of measuring eye tracking are not accurate AND they lose accuracy over time. There is a misalignment between what we are measuring (on which platforms) and what this means for different platforms. Even if accurate, eye tracking measures are not the best-in-class measure for attention. While EEG is a great measure, it is not scalable. Heart rate, on the other hand, can scale. Pilot project—massive in scale—50 programs, n>4000 using Stream Pulse platform; 1,576 in lab sessions and 2,437 in home sessions. Results demonstrate that HR via fitness trackers is viable with correlations as high as .81, but overall, this was only .32. Importantly, while promising, HR is a complex measure that necessitates more work to clean noise and harness full potential. Next steps: 1. Continue mining existing data to devise strategies for identifying and cleaning noise factors. 2. Large pilot “in-the wild” using natural fitness tracker data combined with ACR. Open questions: Where is attention being oriented to? Towards brand? If not, what’s the value in that? Then, thinking about what effect this has. Finally, not all measures lend themselves to AI. Trained against what reference data—is it measuring attention or something else, this depends on the reference data. A lot of the companies are using sales lift measures—you need attention to get sales lift, but you don’t measure this as attention. Finally, with what validation?

Key Takeaways

  • Inattention should be the focus.
  • Eye tracking is not the best measure; heart rate and EEG are better but only heart rate is scalable.
  • Heart rate is viable and scales so there is promise in its application.
  • Be skeptical (but hopeful) of AI measure of attention. Proper scientific validation is necessary before we adopt them.

Download Presentation

Member Only Access

AUDIENCExSCIENCE 2023

The ARF hosted its annual flagship conference, AUDIENCExSCIENCE 2023, on April 25-26, 2023. The industry’s biggest names and brightest minds came together to share new insights on the impact of changing consumer behavior on brands, insights into TV consumption, campaign measurement and effectiveness, whether all impressions are equal, join-up solutions across multiple media, the validity, reliability and predictive power of Attention measures, targeting diverse audiences, privacy’s effect on advertising and the impact of advertising in new formats. Keynotes were presented by Tim Hwang, author of Subprime Attention Crisis, Robert L. Santos of the U.S. Census Bureau, Brian Wieser of Madison and Wall, LLC and Andrea Zapata of Warner Bros. Discovery.

Member Only Access

ATTENTION METRICS: Moving from Laboratory to Field Applications

On September 12, the ARF and Adelaide co-hosted an event to deepen understanding around how attention metrics work, how to utilize them and the nuances of their application to media and advertising quality measurement. Attention Economy experts from Adelaide, Anheuser-Busch InBev, the ARF, The Attention Council, Havas Media Group, OMD USA, Realeyes and TVision hosted shared presentations and held focused discussions on how attention metrics can optimize the match between media placement and consumer targets, improve creative testing and overall campaign outcomes, and more.

Member Only Access