News
ARF Marketing Materials

Members, News Media and Partner Marketers, please visit our brand page to download ARF logos and banners.
ARF Logos & Banners »

FAQ & Contact Us

Our collection of Frequently Asked Questions will answer many of your initial inquiries.
See all FAQs »

Have a question for the ARF? Contact our staff with a specific inquiry.
Contact Us »

ARF and Committed Partners Announce a 90-Day Data Quality Solutions Plan

June 11, 2009

New York, NY – June 11, 2009

The ARF hosted a pivotal Online Research Quality Council meeting that revealed a bold 90-Day Plan to address many of the major issues surrounding online data quality, driven by conclusive research results from the Foundations of Quality study, combined with the wisdom and guidance of industry leaders that are represented on the Council.

Research competitors, clients, and partners were drawn together earlier this week at the ARF by a sense of urgency to reestablish the trustworthiness of online research and advance a shared objective to identify and implement industry-wide solutions. The ARF’s Online Research Quality Council featured detailed findings from an extraordinary US R&D project, called “Foundations of Quality” (FoQ).  This unprecedented effort was enabled by the remarkable cooperation of 17 leading online panel companies and large buyers of marketing research working together in a collaborative and transparent way. 

The meeting produced a sense of urgency that reflected the hard work of the ARF and industry leaders who are contributing their time. This will continue, and commitment to a “90-Day plan” was voiced by the ARF and the ORQC Governing Board.  With the cooperation of other industry associations, the ARF will provide a program with recommendations regarding metrics and business practices, templates, definitions, and training that will enable buyers and sellers to work together to bring the industry to a better place in terms of data quality, comparability, and the trustworthiness of online research results.

The study collected over 100,000 completed survey responses from 17 panels in a rigorously designed research project. The question with the most important implications for the industry that the study addressed is: Why do survey results not replicate? Panels appear to produce different answers, so switching panels or changing the mix of panels can lead to results that are not comparable across studies. Panel practices vary widely, which influences results; for example, in the FoQ study it was found that purchase intent is higher in panels that offer cash incentives.  Such panels also seem to attract more panelists who say they are motivated to take surveys for money. 

“Lack of comparable results across studies was the original smoking gun regarding online data quality, said Joel Rubinson, Chief Research Officer of the ARF, “There are immediate dividends from FoQ for research buyers and suppliers by understanding that switching panels comes with the risk of discontinuity.” 

The study results address a number of critical questions, including:

  • Is there a small group of “professional respondents” on everyone’s panel, doing it for the money, and gaming the system rather than providing thoughtful answers?  The answer is a resounding “no”.  A small proportion of people are on more than one panel, and the panelist pool is not small, in fact comparable or better than mail panels in their heyday. People who are on multiple panels and who take numerous surveys in a month are, on average, better respondents. 
  • What drives good survey-taking behavior? The underlying driver is length of survey.  Shorter surveys produce fewer “bad respondents”. The optimal number of surveys taken is higher than most expected, and those who are motivated by wanting to share their opinions rather than being in it for the cash gifts also tend to give more thoughtful, consistent answers.
  • Are people taking the same survey more than once?  The potential exists, although it is less than initially reported.  The industry must develop operational approaches to ensure that a survey is not taken more than once.

Dr. Tom Evans of ESPN offered encouragement about the next steps of the project: “As the FoQ initiative moves into the next phase, ESPN, as users of online research, is excited to see that the ORQC has committed to moving quickly to address the original study’s results and how they impact what we think we know and what we do not know about online data quality.”

Robert Tomei, Founder and Co-Chair of the ORQC,added: “As we had committed to the industry, the ORQC is now focused on identifying and implementing solutions to the specific issues that drive online research data quality.  Now that the facts are in, the ORQC is working in a collaborative manner across  research buyers , research agencies and other industry associations to develop a series of recommended approaches, metrics, and procedures that will address many of the quality-related issues that have challenged the industry over the last few years.”

The Online Research Quality Council (ORQC) was established in 2007 under the auspices of the Advertising Research Foundation at the behest of advertisers and research companies to address a critical business and industry need: the improvement of online survey quality. The next meeting of the ORQC is scheduled for ARF Members on September 15th, 2009 2:30 PM at the ARF’s NY HQs. In addition to a 90-Day Version 1.0 plan, Knowledge Briefs and Executive Summaries of the results will be available.

Media Contact
Advertising Research Foundation 
Barry Schwartz
212-677-8700 Ext 18
barry@schwartzpr.com

Board of Directors

A veritable “who’s who” of industry leaders have been elected to the Advertising Research Foundation's Board of Directors.
Meet the Board of Directors »

Meet the Staff

See who is hard at work behind the scenes of ARF's conferences, member events, and research initiatives.
Meet the staff »