Research & Data Quality

• Research & Data Quality
• Article

Learn About the Benefits, Limitations and Dangers of Using LLMs in Advertising Research

• By ARF
• Knowledge at Hand

This report offers an abridged version of the cumulative 2025 AI handbook, exploring the benefits, limitations and dangers of generative AI models when employed in various tasks associated with marketing and advertising research. This report sums up the insights from several experiments where ARF researchers tested AI's capabilities in survey design, sentiment analysis, coding open-ended responses, language translation and addressing AI risks.

• Research & Data Quality
• Article

How Ratio Framing Shapes Marketing Decisions

• MSI

Marketers often use ratios like ROAS (Return on Advertising Spend) and ACOS (Advertising Cost of Sales) to communicate campaign efficiency. Researchers in this study find that the way in which these ratios are framed can dramatically influence stakeholder perceptions and decisions. Published in a working paper of the Marketing Science Institute (MSI), the study reveals that even mathematically equivalent ratios can shape contrasting judgments about marketing effectiveness, investment continuation and strategy preferences. This research underscores the psychological power of presentation and calls for smarter framing in marketing metrics.

• Research & Data Quality
• Article

Should Experiments Use LLMs as Human Surrogates? This Study Gives a Resounding No.

This study evaluates the reasoning depth of large language models (LLMs) using the 11-20 Money Request Game, an experimental game designed to test level-k reasoning. Level-k reasoning is a theoretical framework in game theory where individuals operate at varying levels of strategic thinking. The findings of the study reveal significant differences between the responses of LLMs and human participants, highlighting the limitations of using LLMs as human surrogates in behavioral experiments. This research emphasizes the need for caution when interpreting LLM behavior as human-like, as the models often exhibit inconsistent and non-human-like reasoning patterns. The study suggests that while LLMs can provide valuable insights, they should not be relied upon as accurate simulations of human behavior.

• Research & Data Quality
• Article

The Importance of Incrementality in Retail Media Measurement

• INSIGHTS STUDIOS

Despite massive growth driven by significant investments, retail media performance measurement still falls short in many areas. On October 15, OptiMine and Best Buy dove deep into the use of incrementality measurement for retail media, how it works and why it is so unique in the RMN space. Attendees explored why (and how) some of the world’s largest brands have embraced it for improved success.

• Research & Data Quality
• Article

The 2025 Guide on Using AI for Advertising Research

• ARF ORIGINAL RESEARCH

This update to the ARF’s first comprehensive handbook, released last year, provides an exploration into the burgeoning technology’s transformative role in marketing and advertising research. It covers the integration of AI into marketing strategies, ethical considerations, future trends and practical case studies. This handbook is an essential guide for advertising research professionals looking to leverage any of the latest AI platforms while ensuring ethical and impactful outcomes.

• Research & Data Quality
• Article

The State of Privacy-First Marketing & Redefining Performance

• Insights Studio Series

How we use data today looks different given evolving regulations, platform changes, consumers expectations of data transparency, and more. Our Insights Studio on January 30 explored the latest developments in data privacy and how they are impacting marketing strategies. Panelists unveiled strategies to establish consumer trust and effectively market, while aligning with privacy regulations.

• Research & Data Quality
• Article

AI for Everyday Use

On November 20, the ARF held a workshop exploring prompts, personas and how to use AI responsibly. This dynamic event, designed for advertising and marketing professionals looking to explore the evolving landscape of AI-powered research, provided insights into prompt crafting. Participants also gained a deeper understanding of the promises and pitfalls of using personas in AI-powered advertising and marketing research.

• Research & Data Quality
• Article

LOLA: Revolutionizing Content Experiments with LLM-Assisted Online Learning

In the rapidly evolving digital content landscape, media firms and news publishers require automated and efficient methods to enhance user engagement. This study introduces the LLM-Assisted Online Learning Algorithm (LOLA), a novel framework that integrates Large Language Models (LLMs) with adaptive experimentation to optimize content delivery. Leveraging a large-scale dataset from Upworthy, which includes 17,681 headline A/B tests, the study investigates three pure-LLM approaches and finds that prompt-based methods perform poorly, while embedding-based classification models and fine-tuned open-source LLMs achieve higher accuracy.

LOLA combines the best pure-LLM approach with the Upper Confidence Bound (UCB) algorithm to allocate traffic and maximize clicks adaptively. Numerical experiments on data from the website Upworthy show that LOLA outperforms the standard A/B test method, pure bandit algorithms and pure-LLM approaches, particularly in scenarios with limited experimental traffic. This scalable approach is applicable to content experiments across various settings where firms seek to optimize user engagement, including digital advertising and social media recommendations.