The noisy gatekeeper: Measuring the reliability of peer review and testing a way to improve it, with Felix Holzmeister

Please join us for the sixth AIMOS Metascience Seminar.

Speaker: Felix Holzmeister

Summary: Virtually everything science “knows” has passed through peer review. It decides what gets published and - downstream - who gets believed, funded, and hired. Yet while open science has transformed how we report research, research evaluation has largely escaped the same scrutiny. So how well does science’s gatekeeper actually work, and could it work better? This talk presents two complementary attempts to answer these questions. The first is a diagnosis at unprecedented scale: drawing on 11.7 million reviews of 4.7 million manuscripts at more than 1,900 Elsevier journals, it asks how often reviewers of the same manuscript actually agree. The second is a treatment: a preregistered field experiment within a journal’s editorial process that redesigns what reviewers are asked to produce. Together, the studies put numbers on questions usually argued from anecdote: how much of peer review is signal and how much is lottery, what reviewer disagreement means for the false-positive and false-negative rates of editorial decisions, and which levers could make the gatekeeper better at its job. Some of the answers are reassuring; several are not.

Host: Jan Feld and AIMOS