One technique that offers a treasure trove of insights into consumer behavior is “web scraping.” Although worthwhile, using it to gather data requires a specific, methodological approach. Otherwise, validity is threatened. This MSI working paper addresses how researchers should go about web scraping in order to ensure design transparency, analytic reproducibility, analytic robustness, replicability and generalizability of effects.
View article.