Many Labs 2 : Investigating Variation in Replicability Across Sample and Setting

Klein, RA and et al, 2018, 'Many Labs 2 : Investigating Variation in Replicability Across Sample and Setting' , PsyArXiv Preprints .

PDF (Published under Creative Commons CC0 1.0 Universal licence) - Published Version
Download (4MB) | Preview


We conducted preregistered replications of 28 classic and contemporary published findings with protocols that were peer reviewed in advance to examine variation in effect magnitudes across sample and setting. Each protocol was administered to approximately half of 125 samples and 15,305 total participants from 36 countries and territories. Using conventional statistical significance (p < .05), fifteen (54%) of the replications provided evidence in the same direction and statistically significant as the original finding. With a strict significance criterion (p < .0001), fourteen (50%) provide such evidence reflecting the extremely high powered design. Seven (25%) of the replications had effect sizes larger than the original finding and 21 (75%) had effect sizes smaller than the original finding. The median comparable Cohen’s d effect sizes for original findings was 0.60 and for replications was 0.15. Sixteen replications (57%) had small effect sizes (< .20) and 9 (32%) were in the opposite direction from the original finding. Across settings, 11 (39%) showed significant heterogeneity using the Q statistic and most of those were among the findings eliciting the largest overall effect sizes; only one effect that was near zero in the aggregate showed significant heterogeneity. Only one effect showed a Tau > 0.20 indicating moderate heterogeneity. Nine others had a Tau near or slightly above 0.10 indicating slight heterogeneity. In moderation tests, very little heterogeneity was attributable to task order, administration in lab versus online, and exploratory WEIRD versus less WEIRD culture comparisons. Cumulatively, variability in observed effect sizes was more attributable to the effect being studied than the sample or setting in which it was studied.

Data, materials and code available at:

Item Type: Article
Contributors: Coen, S (Contributor)
Schools: Schools > School of Health and Society
Journal or Publication Title: PsyArXiv Preprints
Related URLs:
Depositing User: USIR Admin
Date Deposited: 21 Nov 2018 12:57
Last Modified: 16 Feb 2022 00:19

Actions (login required)

Edit record (repository staff only) Edit record (repository staff only)


Downloads per month over past year