Social Sciences Replication Project

About the Project

Why are we doing this?
There has been an increasing interest in the predictors of reproducibility of research results, and how low reproducibility may inhibit efficient accumulation of knowledge. The best way to gain better understanding of reproducibility is to study it by trying to replicate published results.

What exactly are we doing?
We will replicate 21 experimental studies in the social sciences published in Nature and Science in 2010-2015. These papers were objectively chosen because they were published in these two high-profile journals in this time period, they share a common structure in testing a treatment effect within or between subjects, they test at least one clear hypothesis with a statistically significant finding, and they were performed using students or accessible convenience samples. We plan to conduct the replications between September 2016 and September 2017.

There will be associated prediction markets:
We have identified one critical finding as a specific statistical focus of replication in each paper. We focus on a single finding because a prediction market (PM) will be conducted after the details of planned replication are determined. The PMs will provide a collective forecast of the chance that a particular target result will replicate (as defined precisely).

Replication procedure:
The replications will be carried out in a two-step process. We will carry out a first data collection based on having 90% power to detect 75% of the effect size reported in the original study (with the effect size transformed into the correlation coefficient, r, to have a common standardized effect size measure across all studies; see Open Science Collaboration (Science, 2015) and Camerer et al. (Science, 2016) for further details about the estimation of r). The criteria for replication is an effect in the same direction as the original study and a p-value<0.05 (in a two-sided test). If the result is replicated in the first data collection, no further data will be collected. If the result is not replicated in the first data collection, a second data collection will be carried out based on having 90% power to detect 50% of the original effect size in the pooled sample of the first and second data collection. If a second data collection is carried out, it will be tested if the original result replicates in the pooled sample of the first and second data collection.