Scaling Laws for Reward Model Overoptimization

In reinforcement learning from human feedback, it is common to optimize against a reward model trained to predict human preferences. This research studies how the gold reward model score changes as we optimize against the proxy reward model using either reinforcement learning or best-of- sampling.

This research can help businesses in the field of AI alignment with reinforcement learning from human feedback. It provides insights into the effects of optimizing against a reward model trained to predict human preferences.