Pretraining Language Models with Human Preferences

Pretraining with human feedback results in much better preference satisfaction than standard LM pretraining followed by finetuning with feedback.

Implementing conditional training, or learning distribution over tokens conditional on their human preference scores given by a reward model, can reduce the rate of undesirable content by up to an order of magnitude when generating without a prompt and with an adversarially-chosen prompt. Pretraining with human feedback results in much better preference satisfaction than standard LM pretraining followed by finetuning with feedback.