Stop Automating Peer Review Without Rigorous Evaluation

Abstract

Large language models offer a tempting solution to address the peer review crisis. This position paper argues that today's AI systems should not be used to produce paper reviews. We ground this position in an empirical comparison of human- versus AI-generated ICLR 2026 reviews and an evaluation of the effect of automated paper rewriting on different AI reviewers. We identify two critical issues: 1) AI reviewers exhibit a hivemind effect of excessive agreement within and across papers that reduces perspective diversity. 2) AI review scores are trivially gameable through paper laundering: prompting an LLM to rewrite a paper could significantly increase the scores from AI reviewers, demonstrating that LLM reviewers are easy to game through stylistic changes rather than scientific results. However, non-gameability and review diversity are necessary but not sufficient conditions for automation. We argue that addressing the peer review crisis requires a science of peer review automation -- not general-purpose LLMs deployed without rigorous evaluation.

The authors' own abstract, copied from arxiv.org.

How to cite it

Built from the details listed on this page. Check them against the paper before you submit.

Baumann, J., Pei, J., Koyejo, S., & Hovy, D. (2026). Stop Automating Peer Review Without Rigorous Evaluation. ICML 2026 — Position Paper Track (Oral). https://arxiv.org/abs/2605.03202

Baumann, Joachim, et al. "Stop Automating Peer Review Without Rigorous Evaluation." ICML 2026 — Position Paper Track (Oral), 2026, https://arxiv.org/abs/2605.03202.

@article{baumann2026stop,
  title = {Stop Automating Peer Review Without Rigorous Evaluation},
  author = {Joachim Baumann and Jiaxin Pei and Sanmi Koyejo and Dirk Hovy},
  journal = {ICML 2026 — Position Paper Track (Oral)},
  year = {2026},
  url = {https://arxiv.org/abs/2605.03202},
}

What it is

Peer-reviewed ICML 2026 position paper by Joachim Baumann, Jiaxin Pei, Sanmi Koyejo, and Dirk Hovy, accepted for an oral presentation. Compares human and AI ICLR 2026 reviews and tests automated paper rewriting, reporting excessive reviewer agreement and susceptibility to stylistic changes. The authors argue for rigorous evaluation before automating peer review.

Topics