The Power and Pitfalls of Megastudies for Advancing Applied Behavioral Science

Increasingly, policymakers are turning to behavioral science for insights about how to improve citizens’ decisions and outcomes. However, these insights can only inform policy insofar as they are comparable—and unfortunately, different intervention ideas are typically tested across different samples on different outcomes over different time intervals. In my talk, I will introduce the “megastudy,” a massive field experiment in which the effects of many different interventions are compared in the same population on the same objectively measured outcome for the same duration. I will then share results from four megastudies that my team has conducted over the last several years to illustrate the power and pitfalls of this methodology. I will describe a megastudy encouraging more gym visits (N=61,293 24 Hour Fitness members), a megastudy encouraging flu vaccination at doctor’s appointments (N=47,306 patients of Penn Medicine and Geisinger Health), a megastudy encouraging in-pharmacy flu vaccinations (N=689,693 Walmart pharmacy patients), and a megastudy encouraging in-pharmacy COVID-19 bivalent booster vaccinations (N=3.5 million patients of a national pharmacy chain). I will discuss the accuracy of experts’ and laypeople’s forecasts of the performance of different interventions tested in these studies, share best practices for running megastudies, and describe some key limits of this approach to applied behavioral science research.