How It Works
Reward Modeling
Training a separate model to predict which of two answers a person would prefer, so it can score new answers automatically. People compare pairs of responses, and the reward model learns to imitate their choices; that score is then used to steer the main model during reinforcement learning. Because the main model is optimized to please the scorer, it can learn to exploit the scorer's blind spots, a problem known as reward hacking.
Origin
Learning a reward function from human comparisons was described in “Deep Reinforcement Learning from Human Preferences” by Paul Christiano, Jan Leike and colleagues (OpenAI and DeepMind, 2017). Jan Leike and co-authors used the term “reward modeling” as the name of a research direction in “Scalable agent alignment via reward modeling” (DeepMind, 2018).