RE-Bench: Evaluating Frontier AI R&D Capabilities (METR)

What it is

METR's November 2024 release of RE-Bench, a benchmark comparing frontier model agents with human experts on seven machine-learning research-engineering tasks. It includes data from 71 human expert attempts and results for Claude 3.5 Sonnet and o1-preview.

Topics