Have a myARF account?

ARF West

Maximizing the Mix: Day 1

Maximizing the Mix: Marketing Performance Measurement in Today’s Media Environment

This is a summary of discussions from Day 1 of The ARF event in San Francisco recapping some of the leading approaches on marketing performance measurement –with particular focus on mobile social and cross-platform.

Gayle Fuguitt, The ARF CEO, kicked off top-lining the simplicity of the C-Suite ask

  • Do my marketing dollars drive growth?
  • Where do I spend my next marketing dollar?

The ARF’s goal is helping network our way to success by getting the solutions out and having candid conversations about them.

Using Causality to Maximize Profits Christie Blomster, Product Marketing Manager — Facebook

Changing media landscape, changing consumer behavior and use of multiple devices are making it more complex and more difficult than ever to measure effectiveness.


This is even more true for digital where we have built attribution models from

  • Last touch gets all the credit
  • First touch gets all the credit
  • First/Last touch % shared credit
  • Tiered credit given to most valued touch
  • Sophisticated weighting of touches on propensity to purchase
  • Time decay statistical significance models


But all these models, algorithms and approaches suffer from their foundation in “observable behavior” and a model is only as good as its data inputs. But observational data cannot determine causality and you need causal difference for better decisions.

We need to get as close to the Ground Truth as possible. To get there we cannot just measure “did my ad reach people likely to convert and draw a correlation?” Instead, we need to know did my ads cause people to convert/act.

To do that we need to know what would have happened if we did not runs ads. A base or “counter factual” that tells us what sales levels would have been acheived even without advertising is crucial. Sounds easy . . . but it’s not.

A key problem is when we implement a typical test and control model, it is common to compare Brand messaging to one group vs. a generic (e.g PSA/placebo) messaged to a control group. In today’s digital world with the dominance of programmatic placement a “true control” is more elusive than ever. The machine learning of our ad delivery engines are smarter than our data sets.

Facebook ran a campaign where the control was a “Smokey the Bear” ad versus the Tested. However, instead of true random samples the delivery algorithms maximized audience for both and “Smokey the Bear” was sent to so many well targeted environmentalists and great reach that led to negative lift, where the Test where results demonstrated “no advertising” drove more sales than supported efforts. Machine learning was so good that there was not a clean control.

How do we solve this? We need better randomized control experiments on an Rx testing model. What Facebook calls the “scale of truthiness” and test groups need to be on the TRUTH end of that scale.

Because Facebook knows the individuals, they can uniquely actually pull sample audiences that are truly randomized. This allows causal inference from delivering ads to just the test group, not the control group and compare the KPI and lift. The hallmark is true randomization across both observable and non-observable variables.

Specifically identifying “responsive audiences” a segmentation based on characteristics of people who converted vs. those who would not have converted without the advertising/stimulus and improve the campaign ROI. However, with the right data and the right measurement you can accelerate the growth from added marketing support and understand the best messaging and segmentation factors.


Return on Word of Mouth (WOM)

Nancy Smith – President & CEO, Analytic Partners

Rob Key – CEO, Converseon

Ed Keller – CEO, Keller Fay Group

Alice Sylvester – Partner, Sequent Partners

Led by WOMMA (Word of Mouth Marketing Association, this is original research sponsored by AT&T, Discovery, Intuit, Pepsico and WeightWatchers.


Marketers need to measure social and word of mouth to be able to invest in it. We know that there’s “nothing more valuable than the recommendation from a friend” but how much to invest to drive that.


This was original research supported by an extensive data collection effort. 2-3 years of weekly data on Word of Mouth (WOM)

  • Business performance (sales)
  • Offline WOM (provided by Keller Fay Group)
  • Online WOM (provided by Converseon)
  • Paid media by channel
  • Promotional Activity
  • Other external and internal factors


Data was scored for modeling and accuracy

Key findings:

  • Total WOM impact was significant on all businesses, ranging from 3% to 30%, with 13% impact on average.  Higher price / higher consideration categories were impacted more by WOM than lower price / lower consideration categories.
  • On average, offline WOM drove 2/3 of the lift and online WOM about 1/3.  There were also differences in the impact of positive and negative WOM, with the latter having a more powerful impact.
  • In terms of how WOM works, it’s not simply a question of direct impact but also one of an “amplifier” effect.  Structured equation modeling (SEM) explored this in detail and highlighted the interaction between WOM and paid media (including search).
  • A key finding was also time to impact: WOM had a more immediate impact than traditional advertising.


For generations there’s been an understanding that WOM was important but it was like “qualitative” learning. The importance of this study is giving WOM and social a stronger “quantitative” base to support relative value and predictive modeling.

To the participants one of the next steps is the impact of the creative element and learning what creative more strongly drives synergy with WOM. Another next step will be expanding the categories and that this methodology is applied to.


Keep Up With the Fastest-Growing Channels: Expert Perspectives

Christi Eubanks – Senior Manager, Global Digital Analytics, Mattel

Sable Mi – Senior Director, Research Strategic Planning, BrightRoll

A wish list item from Mattel, having platforms move their pitch from “how cool the platform is” and “how rich the engagement” to the value of one platform vs. another and how likely to impact sales. From C. Eubanks perspective, “finding that common benchmark where rich data from Facebook can be balanced against good data from YouTube vs. tolerable date from Instagram”.

S. Mi emphasizes the importance of clearly defining your goal and KPI upfront. Are you buying digital VOD to extend reach, drive brand awareness or influence intent? How does the content of the digital VOD compare to the linear TV message? Are you testing the impact of digital VOD or one message vs. an alternative?


G. Fuguitt asked the opinion of the state of mobile measurement and the state of social measurement from 1 to 10 with 1 being far away from ideal and 10 being well understood.

  • Mobile we’re at 7.5. Doing a good job with strong analytics packages with deep information. In mobile apps you can “own” your user. Message that user.
  • Social further back. We’re “renting” the user. Learning is immature across the board. The big platforms have good data but very little info on organic data, so vital to overall social impact.

Measurement of Digital Consumers’ Multi-Screen Behavior: Understanding the Right Touch Points

Hannu Verkasalo, Ph.D. – CEO, Verto Analytics, Inc.

Looking at the major gaps in measurement of mobile, apps, wearable and smart TVs is the gap between the time consumers spend on each device/channel and the percent of advertising dedicated to that device/channel. The largest gap being digital and mobile with 50% of consumer time and less than 30% of the spend.


Verto recommends understanding and assigning communication role to each device/channel

  • Mobile: go-to-action (location-based messaging and 1:1 text ads)
  • Web/Desktop: consideration tool (drive to search, aid product research)


And the importance of time of day and behavior. Facebook for example during daytime use is a very single screen activity with independent communication impact. In the evening Facebook usage becomes more multi-screen driving interactive and   co-dependent messaging outcomes.


Link to posted video:


Link to Re:Think registration (3/16-18 in NYC):