Improving the alignment of LLMs to preferred outcomes is the focus of a study by Arka Pal et al. (2024), introducing Direct Preference Optimisation Positive (DPO-Positive).

• By fine-tuning with DPO-Positive, researchers have improved LLM performance across numerous downstream tasks.
• This comes after discovering that the standard DPO loss could potentially reduce a model’s preference for certain examples under specific conditions.
• New model versions like Smaug-34B and Smaug-72B have now achieved state-of-the-art performance, with notable improvements in open-source LLMs.
• This paper marks a significant stride in preference-based model fine-tuning and will broadly impact the development of more responsive and accurate language models.