AI and the Everything in the Whole Wide World Benchmark

Abstract

There is a tendency across different subfields in AI to valorize a small collection of influential benchmarks. These benchmarks operate as stand-ins for a range of anointed common problems that are frequently framed as foundational milestones on the path towards flexible and generalizable AI systems. State-of-the-art performance on these benchmarks is widely understood as indicative of progress towards these long-term goals. In this position paper, we explore the limits of such benchmarks in order to reveal the construct validity issues in their framing as the functionally "general" broad measures of progress they are set up to be.

The authors' own abstract, copied from arxiv.org.

How to cite it

Built from the details listed on this page. Check them against the paper before you submit.

Raji, I. D., Bender, E. M., Paullada, A., Denton, E., & Hanna, A. (2021). AI and the Everything in the Whole Wide World Benchmark. Preprint, not yet peer-reviewed. https://arxiv.org/abs/2111.15366

Raji, Inioluwa Deborah, et al. "AI and the Everything in the Whole Wide World Benchmark." Preprint, not yet peer-reviewed, 2021, https://arxiv.org/abs/2111.15366.

@article{raji2021ai,
  title = {AI and the Everything in the Whole Wide World Benchmark},
  author = {Inioluwa Deborah Raji and Emily M. Bender and Amandalynne Paullada and Emily Denton and Alex Hanna},
  journal = {Preprint, not yet peer-reviewed},
  year = {2021},
  url = {https://arxiv.org/abs/2111.15366},
}

What it is

Raji and co-authors show that benchmarks claiming to measure general ability measure something much narrower, and that the gap is how overclaiming happens.

Topics