Large Language Models (LLMs) continued their important role in 2024, with several major developments completely outperforming previous models. The focus continued to more smaller, more powerful models from companies like Meta, Qwen, or Google. These models not only became more powerful, but also more efficient.
\nPart of the reason is the continued scaling of post-training methods and datasets. Post-training methods like Direct Preference Optimization (DPO), Proximal Policy Optimization (PPO), and Group Preference Optimization (GRPO) have been used to align models with human preferences and improve performance.
\nContinuing from How to fine-tune open LLMs in 2025 with Hugging Face, this guide focuses on aligning models using Direct Preference Optimization (DPO). We'll build upon our previously fine-tuned model from the Fine-Tune LLMs in 2025 guide and improve it further through preference learning.
\nBefore we start, let's take a look at the Direct Preference Optimization (DPO) paper and understand how it works.
\nWhat is Direct Preference Optimization (DPO)?
\nDirect Preference Optimization (DPO) is a simplified approach to align language models with human preferences. Unlike traditional RLHF methods that require training a separate reward model and using complex PPO optimization, DPO treats the alignment problem as a classification task on preference data. This makes it more stable, efficient, and computationally lightweight compared to alternatives.
\nOverview
\nIn this guide, we'll use offline DPO with on-Policy data. Offline DPO is the more compute-efficient, meaning we don't need a Reward model during training in memory. However, we will be staying on-policy. We'll first generate samples from the Supervised Fine-Tuned (SFT) model we trained in the previous guide and score them with a rule-based reward model that checks answer correctness. These scores will then be used to create preference pairs (a "preferred" and a "rejected" response) for DPO training.
\nIf you are going to adjust the example to your own use case, which cannot be verified with a rule-based reward model, you can use a generic reward model like nicolinho/QRM-Llama3.1-8B-v2, which ranks in the top 10 on reward bench or LLM as judge based on your own criteria.
\nYou will learn how to:
\nDPOTrainerNote: This guide is designed for consumer GPUs (24GB+) like the NVIDIA RTX 4090/5090 or A10G, but can be adapted for larger systems.
\nOur first step is to install Hugging Face Libraries and Pytorch, vllm, and trl, transformers and datasets. If you haven't heard of trl yet, don't worry. It is a new library on top of transformers and datasets, which makes it easier to fine-tune, rlhf, align open LLMs.
\nPython
\n# Install Pytorch & other libraries, make sure to match your GPU driver version\n% --- :////\n \n# Install flash-attn\n% - \n \n# Install Hugging Face libraries\n% -- \\\n \\\n \\\n \\\n \\\n \\\n \\\n \nNote: you may need to restart the kernel to use updated packages.
\nWe will use the Hugging Face Hub as a remote model versioning service. This means we will automatically push our model, logs and information to the Hub during training. You must register on the Hugging Face for this. After you have an account, we will use the login util from the huggingface_hub package to log into our account and store our token (access key) on the disk.
Python
\n\n \n # ADD YOUR TOKEN HERE\nWe are going to generate a preference dataset from our SFT model rather than using a different model and off-policy data. On-policy data is generated from the model being trained, which means the preference pairs directly reflect the model's current capabilities and output distribution. This makes the training more relevant and effective.
\nStudies have shown that leveraging on-policy preference data leads to better performance compared to off-policy approaches. REF 1, REF 2.
\nAs we are building on our previous fine-tuned model we want to furhter improve the performance on MATH related tasks. We will use the prompts from philschmid/DMath. DMath (Diverse Math Word Problems) is a collection of 10K high-quality grade school-level math word problems for the paper "It Ain't Over: A Multi-aspect Diverse Math Word Problem Dataset". Each solution will be scored by a rule-based reward model that checks answer correctness using a regex. (Thats a very simple and naive approach, better would to additionally use specific format extraction, e.g. "The Answer is: [ANSWER]" or additional LLM Judge). If the solution is correct, it will be labeled as preferred, otherwise as rejected. Based on the score we will create difference preference pairs.
\nI prepared create_preference_dataset.py script to generate the dataset. The dataset will be saved to the Hugging Face Hub. We are going to generate 4 solutions for each input hoping to have at least one correct solution and one incorrect solution. If we don't have a pair we will skip the input.
\nPython
\n // -- / -- 5000 -- /-3-1-8b----10k- -- 4 -- 16 \nNote: You can skip this step and used the generated preference dataset from me philschmid/philschmid-llama-3-1-8b-math-orca-spectr-philschmid-DMath-candidates. It includes 1.9k prefence pairs.
\nTRL supports the DPO through a dedicated DPOTrainer for aligning LLMs from preference data, as described in Direct Preference Optimization: Your Language Model is Secretly a Reward Model. The DPOTrainer is a subclass of the Trainer from the transformers library and supports all the same features, including logging, checkpointing, distributed training, and parameter efficient fine-tuning (PEFT). Direct Preference Optimization (DPO) aligns LLMs with human preferences using 'prompt', 'chosen', and 'rejected' columns, where each 'prompt' is paired with a 'chosen' response that aligns with human preferences and a 'rejected' response that does not.
We prepared a run_dpo.py scripts, which supports providing a yaml configuration file. This allows you to easily change the model, dataset, hyperparameters, and other settings. This is done by using the TrlParser, which parses the yaml file and converts it into the TrainingArguments arguments.
Based on the Alignment Handbook we know that we need to use a ~10-100x smaller learning rate for DPO compared to SFT. In our example we reduce the learning rate from 2e-4 (SFT) to 5e-6 (DPO). Another important parameter is the beta parameter, which is used to control the strength of the alignment, typically something in the range of 0.1 to 0.5. A higher beta means less divergence from the initial reference model or the text generations are very similar in terms of their probability distributions.
The script is kept very simple and is easy to understand. This should help you understand, customize and extend the script for your own use case. We define dataclasses for our arguments. Every argument can then be provided either via the command line or by providing a yaml configuration file. That way we have better type safety and intellisense support.
Python
\n# ....\n \n\n\n : \n ...\n# ....\nThe training script is separated by ####### blocks for the different parts of the script. The main training function:
prompt, chosen and rejected columnBelow is an example recipe of how we train Llama 8B model with DPO and Q-LoRA.
\nYAML
\n# Model arguments\nmodel_name_or_path: philschmid/llama-3-1-8b-math-orca-spectrum-10k-ep1\ntokenizer_name_or_path: philschmid/llama-3-1-8b-math-orca-spectrum-10k-ep1\nmodel_revision: main\ntorch_dtype: bfloat16\nattn_implementation: flash_attention_2\nuse_liger: false\nbf16: true\ntf32: true\noutput_dir: runs/dpo-llama-3-1-8b-math-ep3\n \n# Dataset arguments\ndataset_id_or_path: philschmid/philschmid-llama-3-1-8b-math-orca-spectr-philschmid-DMath-candidates\n \n# LoRA arguments\nuse_peft: true\nload_in_4bit: true\nlora_target_modules: "all-linear"\n# important as we need to train the special tokens for the chat template of llama \nlora_modules_to_save: # you might need to change this for qwen or other models\nlora_r: 16\nlora_alpha: 16\n \n# Training arguments\nbeta: 0.1\nmax_length: 1536\nmax_prompt_length: 768\nloss_type: sigmoid # default loss, alternatives: https://huggingface.co/docs/trl/dpo_trainer#loss-functions\nnum_train_epochs: 3\nper_device_train_batch_size: 1 \ngradient_accumulation_steps: 8\ngradient_checkpointing: true\ngradient_checkpointing_kwargs:\n use_reentrant: false\nlearning_rate: 5.0e-6 \nlr_scheduler_type: constant\nwarmup_ratio: 0.03\n \n# Logging arguments\nlogging_strategy: steps\nlogging_steps: 5\nreport_to:\n- tensorboard\nsave_strategy: "epoch"\nseed: 42\n \n# Hugging Face Hub \npush_to_hub: true\n # hub_model_id: llama-3-1-8b-math-orca-qlora-10k-ep1 # if not defined same as output_dir\nhub_strategy: every_save\nThis config can be used for single-GPU training (~24GB GPU Memory), if you have more memory available you can increase the per_device_train_batch_size and for multi-GPU training with DeepSpeed (see Appendix for full
_Note: I ran the script on 1x H100 with a batch size of 8 and 8 gradient accumulation steps. This took ~1 hours to complete._command).
\nPython
\n! // -- /--3-1-8b-\nNote: During the training we want to minimize loss and grow reward/margins metrics. Keep an eye on the reward/margins metrics, if they are not growing you might need to increase the beta parameter or adjust the learning_rate.
\nUsing Q-LoRA only saves the trained adapter weights. If you want to use the model as standalone model, e.g. for inference you might want to merge the adapter and base model. This can be done using the following command:
\nPython
\n! / -- /--3-1-8b-- -- True -- --3-1-8b---\nAfter the training is done we want to evaluate and test our model. Similar to our SFT model, we will evaluate the model on GSM8K dataset to see if it improved performance. GSM8K (Grade School Math 8K) is a dataset of 8.5K high quality linguistically diverse grade school math word problems. The dataset was created to support the task of question answering on basic mathematical problems that require multi-step reasoning.
\nEvaluating Generative AI models is not a trivial task since 1 input can have multiple correct outputs. If you want to learn more about evaluating generative models, check out:
\nWe are going to use Evaluation Harness an open-source framework to evaluate language models on a wide range of tasks and benchmarks. The frameworks support evaluating models behind OpenAI compatible API endpoints, those can be locally or remotely. This super helpful as we can evaluate our model in the same environment we will use for production.
\nWe are going to use Text Generation Inference (TGI) for testing and deploying our model. TGI is a purpose-built solution for deploying and serving Large Language Models (LLMs). TGI enables high-performance text generation using Tensor Parallelism and continous batching. If you are or want to use vLLM you can check the Appendix on how to start the inference server.
\nNote: Make sure that you have enough GPU memory to run the container. Restart kernel to remove all allocated GPU memory from the notebook.
\nWe will start the on 1 GPU detached. Meaning we can can continue to use the notebook while the container is running. If you have more GPUs you can change the --gpus and --num-shard flags to the number of GPUs.
Bash
\n\n \nnum_gpus=1\nmodel_id=philschmid/dpo-llama-3-1-8b-math-ep3-merged # replace with your model id\n \n\n \nOur container will now start in the background and download the model from Hugging Face Hub. We can check the logs to see the progress with docker logs -f tgi.
Once our container is running we can send requests using the openai or huggingface_hub sdk. Here we ll use the openai sdk to send a request to our inference server. If you don't have the openai sdk installed you can install it using pip install openai.
Python
\n\n \n# create client \n = \n \n = \n \n = \n = \n \n \n# Take a random sample from the dataset and remove the last message and send it to the model\n \n = \n = ..\n \n# Print results\n\n\n\n \nAwesome that looks great! Now we can evaluate our model with the Evaluation Harness.
\nNote: Make sure to change the model id to your fine-tuned model.
\nPython
\n! -- -- \\\n -- \\\n -- =/--3-1-8b---,=://:8080///,=8,=10,=False,=180,=4096 \\\n -- \\\n --\nWow, 59% accuracy, thats a 5% improvement from our SFT model, using only ~2k preference pairs for 3 epochs. That shows that our script and config is working correctly.
\nNote: You might be able to achieve better results with more data, more epochs or tuning the hyperparameters (beta, learning rate, batch size, etc.). I ran some ablations on multi-gpu training and full training with DeepSpeed (see Appendix for full command) and the best results was 62% accuracy.
\nPython
\n! \n! \nIn this guide, we successfully aligned a Llama 3.1 8B model using Direct Preference Optimization (DPO) and achieved a 5% improvement on GSM8K compared to our SFT baseline. Instead of using off-policy preference data (like existing human preference datasets), we generated our own preference pairs using the SFT model we wanted to align. This approach has several advantages:
\nWe also learned that even with a relatively small dataset (~2k preference pairs) and just 3 epochs of training, DPO can produce meaningful improvements in model performance. The combination of on-policy data generation and DPO provides a practical and efficient approach to aligning language models with specific objectives. This method could be adapted for various domains beyond math problems by using different reward models or evaluation criteria to generate preference pairs.
\nNote: Make sure to install deepspeed and accelerate before running the commands. pip install deepspeed==0.15.4
Bash
\nACCELERATE_LOG_LEVEL=info \n