- Past Event Highlights
- Article
A Clean Room Incrementality Experiment – An Indeed Case Study
Joe Zucker – Senior Manager, Marketing Analytics, Indeed
Clean room experiments are challenging in an online marketplace, such as Indeed’s job site for employers and employees, due to potential online experimentation biases, including activity bias, ad server bias and base rate bias, according to Joe Zucker (Indeed). Control groups can be created in multiple ways with different degrees of technical setup or in some cases, external modeling. The five variations of control groups are ghost ads, publisher house ads, PSA ads, propensity score matching and intent to treat. A comparison indicated that each option has both pros and cons, including cost, the need for additional data or publisher support. Joe reminded the audience that there is “no free lunch.” Ghost ads would be preferred by Indeed to create the control group; however, this option has high technical set-up requirements, few publisher partners have this capability and there is low control over the analysis. There are also challenges related to interpreting experimental results, which include low match/conversion rates and the need to analyze experiments with different control group construction. Indeed was able to measure aggregate incrementality for their campaign metrics and prove the value of their advertising as a result of these clean room experiments. Key takeaways:- Despite the challenges of clean room experiments, these experiments are critical to the measurement of the incremental impact of advertising on KPIs.
- Clean room experiments can ensure high quality continuous reporting with actionable analytics and insights while achieving user data privacy compliance.
- Experimentation enabled Indeed to focus on new customers in a cookie-free, privacy-forward manner with the ability to verify advertiser data.
