Summary
This article investigates to what degree a modelled approach to non-probability sample screening and weighting improves data accuracy.
Using the ARF’s Foundations of Quality survey data, the authors demonstrate that by applying optimization methods to a large, diverse data set, several key screening variables were identified. They minimized bias (for more than 100 questions covering 14 substantive areas) across 17 sample sources and two sample types.
The evidence suggests that some or all of the following nonattitudinal variables, if included in an adjustment model, can reduce bias by 23 percent beyond the age/gender/race baseline case:
- Time Spent Online
- Race-Ethnicity
- Education
- Income
- Housing Status
- Political Party
- Presence of a Landline Telephone
- Number of Adults in the Household
- Number of Vehicles in the Household
The bias reduction can increase to 35 percent by adding to the model some or all of the following attitudinal variables:
- “Hopeful”: How often do you feel hopeful?
- “Privacy Concerned”: How concerned are you about having records containing your personal information stolen over the Internet?
- “Open Minded”: How much do you agree with the following statement: ‘It is best to treat those who disagree with you with leniency and an open mind as they may be proven right’?
- “Optimistic”: How much do you agree with the following statement: ‘I am optimistic about my future’?
The authors’ advice to survey practitioners: take a balanced approach. Begin by including variables such as age, gender, region, and race-ethnicity in the sampling scheme. Then include in the weighting scheme these same variables and some of the socio-demographic and attitudinal ones identified in this paper.