research methods

The ARF Member AI Workshop

The ARF Member AI Workshop

The ARF Member AI workshop introduced members to the potentialities of various AI platforms and tools to boost their work productivity. The workshop covered how LLMs such as Copilot, ChatGPT, Gemini and Claude can be employed in three main areas: presentations and reports, advertising research and meetings. Issues such as privacy and security of using AI, as well as the current limitations and challenges of the technology were also discussed. The hands-on, interactive workshop was an opportunity for all those interested in best practices and guidelines for using AI to learn how to interweave such programs into their daily work processes.

Member Only Access

Research to Improve AI

  • Posted: April 19, 2024

Yes, AI is a great tool for marketers. But how can we avoid the “AI Conundrum” – taking advantage of its strengths while avoiding its errors and risks?    

Read more »

Experimentation Unleashed: Driving Transformation Using Cutting-Edge Data

Experimentation Unleashed: Driving Transformation Using Cutting-Edge Data

Cesar Brea – Partner, Bain & Company

James Slezak – CEO, Swayable

Cesar Brea (Bain & Co.) and James Slezak (Swayable) shared the lessons they learned using and experimenting with RCTs (random controlled trials) in trying to transform organizations by taking advantage of new data technologies. They contributed their experiences with CPG, online, event and retailer clients to best exemplify how organizations need to embrace the process of transformation using experimentation and data. Their resulting experimentation maturity framework outlines important conditions for success. Key takeaways:
  • Orchestration is more important than sophistication—think end to end from problem formulation to alignment on execution and measurement with the CFO. Are the conditions right to have a successful experimentation program? What are the underlying organizational and political dynamics that need to be managed? Does the organization have the right tools to support interpretation and adoption?
  • Work with the data that is going to be useful to the company from practical sources to help decision making.

Download Presentation

Member Only Access
Foundations of Incrementality

Foundations of Incrementality

Sophie MacIntyre – Ads Research Lead, Marketing Science, Meta

Randomized Control Trials (RCTs) are the gold standard for unbiased measurement of incrementality according to Sophie MacIntyre (Meta). However, there are situations where RCTs are not available so Meta explored other methods to improve the measurement of incrementality. Meta’s researchers wanted to know how close they could get to the experimental result by using non-experimental methods. The researchers were unable to accurately measure an ad campaign’s effect with sophisticated observational methods. Additionally, traditional non-experimental models like propensity score matching and double machine learning were difficult to use and resulted in large errors. Sophie presented incrementality as a ladder of options that get closer to measuring true business value as the ladder is ascended. The different rungs of the ladder are based on how well a particular measurement approach can isolate the effect of a campaign from any other factors. This research was undertaken in collaboration with the MMA and analyzed non-incremental models, quasi-experiments with incrementality models and randomized experiments. Meta revealed that incrementality could be achieved with modeling if the research included some RCTs. Using PIE (predictive incrementality by experimentation) estimates for decision making led to results similar to experiment-based decisions. Sophie stated that academic collaborations provide quantitative evidence of the value of incremental methods. Key takeaways:
  • Incrementality matters because it is the foundation of good business decisions and should be the “North Star.”
  • Randomized Control Trial (RCTs) are the gold standard for determining incrementality.
  • Using a significant amount of data and complex models can improve the performance of observational methods but does not accurately measure an ad campaign’s effect.
  • Using Predictive Incrementality by Experimentation (PIE) estimates for decision making leads to results similar to experiment-based decisions.

Download Presentation

Member Only Access
 Next »