← All works
Research paper

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

At a glance

DPO trains language models directly on comparisons of preferred and rejected answers.

In one paragraph

Introduced DPO, a simpler way to train language models from examples of preferred and rejected answers. It learns directly from those comparisons, avoiding the separate reward model and reinforcement-learning stage used in earlier approaches.

Authors 6

  1. Rafael Rafailov
  2. Archit Sharma
  3. Eric Mitchell
  4. Stefano Ermon
  5. Christopher D. Manning
  6. Chelsea Finn

Organizations

Credited for this work at publication.