Skip to main content Link Search Menu Expand Document (external link)
Increasing calorie information uptake with a prenudge, with Thunström, Van ‘t Veld, Shogren, and Nordström

This paper is undergoing a complete revision. I expect to share it again early this Fall (2023)

Expecting to get it: An endowment effect for information, with Thunström, Van ‘t Veld, and Nordström

This paper is undergoing a complete revision. I expect to share it again late this Fall (2023)

Decision error decreases with risk aversion: A replication, with Méndez

This paper is undergoing a final revision. I expect to share again this Fall (2023)

Predicting the replicability of social and behavioural science claims from the COVID19 Preprint Replication Project with structured expert and novice groups 📄, with many people

Abstract. Replication is an important “credibility control” mechanism for clarifying the reliability of published findings. However, replication is costly, and it is infeasible to replicate everything. Accurate, fast, lower cost alternatives such as eliciting predictions from experts or novices could accelerate credibility assessment and improve allocation of replication resources for important and uncertain findings. We elicited judgments from experts and novices on 100 claims from preprints about an emerging area of research (COVID-19 pandemic) using a new interactive structured elicitation protocol and we conducted 35 new replications. Participants’ average estimates were similar to the observed replication rate of 60%. After interacting with their peers, novices updated both their estimates and confidence in their judgements significantly more than experts and their accuracy improved more between elicitation rounds. Experts’ average accuracy was 0.54 (95% CI: [0.454, 0.628]) after interaction and they correctly classified 55% of claims; novices’ average accuracy was 0.55 (95% CI: [0.455, 0.628]), correctly classifying 61% of claims. The difference in accuracy between experts and novices was not significant and their judgments on the full set of claims were strongly correlated (r=.48). These results are consistent with prior investigations eliciting predictions about the replicability of published findings in established areas of research and suggest that expertise may not be required for credibility assessment of some research findings.

📄 PDF 🔍 online appendix 💾 replication package

(click on the boxes to see the abstract)