Title: Standardizing mean differences: A cautionary tale from a rational choice perspective

Abstract: In applied areas of science such as education research or medicine, researchers often report the effect sizes of tested interventions to inform people aiming to decide between these interventions. For continuous outcome variables (such as depression or literacy skills), researchers measure an intervention’s effectiveness – its effect size – using the mean difference, quantifying how much, on average, the intervention changed the measured outcome. Or they measure the intervention’s effect size using the standardized mean difference, quantifying by how many standard deviations the intervention, on average, changed the measured outcome. Specifically, researchers frequently rely on standardized mean differences rather than unstandardized ones when they face measurement uncertainty: when they are uncertain how much change in an outcome of interest observed mean differences represent. Standardizing mean differences allows researchers to still convey and compare the effectiveness of interventions across studies despite such measurement uncertainty. But, as I argue, standardizing mean differences when aiming to inform decision-makers is risky; it risks stripping away information rational decision-makers would care about learning. Can we ever avoid this risk? We can in some, restrictive scenarios, as I also show. When should researchers then report standardized mean differences to inform rational decision-making well? My results allow us to recognize that the answer to this question hinges on several philosophical, empirical and methodological questions (mostly) unrecognized in debates on standardizing mean differences.

The seminar will held online on March 12th at 14:30 (Rome time) on the Microsoft Teams platform, here.