Member Only Access
Summary
As generative AI becomes a central tool for producing marketing content, firms increasingly rely on fine-tuning models using engagement data, such as A/B test results. This MSI working paper argues that optimizing only for “what works” risks reward hacking, clickbait and poor generalization. The authors propose a knowledge-guided alignment framework in which large language models (LLMs) generate and validate hypotheses about why content performs well, and then use this knowledge to guide fine-tuning. Using more than 23,000 A/B-tested news headlines, the study shows that knowledge-guided AI produces higher engagement, avoids clickbait and generalizes better—especially in low-data settings.
