DPO vs RLHF/PPO: when would you pick each?

DPO (Rafailov et al., 2023) reformulates the RLHF objective as a closed-form supervised loss on preference pairs, eliminating the explicit reward model and the RL loop entirely. The loss is roughly -log_sigmoid(beta * [logP(y_w|x) - logP(y_l|x)] - beta_ref * [logP_ref(y_w|x) - logP_ref(y_l|x)]). Benefits: (1) simpler training pipeline, no reward-model drift; (2) more stable than PPO (no value head, no sampling during training); (3) works on smaller compute budgets. Limitations: (1) less flexible when you want multi-objective rewards or RL-with-tools; (2) offline only: cannot do online rollouts. Variants that address limits: IPO (handles noisy preferences more gracefully), KTO (single-rating instead of pairs), GRPO (group-relative policy optimization, popularized by DeepSeek-R1), online DPO. Practical rule of thumb 2025-2026: start with DPO, jump to PPO or GRPO only when online exploration or non-pairwise rewards matter.