{ "@context": "https://schema.org", "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-09-03-fine-tuning-a-350m-model-for-better-structured-outputs-in/#webpage", "@type": [ "WebPage", "ArchiveComponent" ], "articleBody": "This guide is a fully public, inexpensive recipe for making a small model substantially better at structured-output compliance. We fine-tune [LFM2.5-350M](https://huggingface.co/LiquidAI/LFM2.5-350M) with Group Relative Policy Optimization (GRPO) using the [TRL library](https://huggingface.co/docs/trl/en/index) and evaluate it on the [IFStruct benchmark](https://huggingface.co/datasets/LiquidAI/ifstruct-v1.0). The full run takes around 500 samples and 100 training steps, small enough for a free-tier Colab or Kaggle GPU, and is available on [GitHub](https://github.com/Liquid4All/cookbook/blob/main/finetuning/notebooks/grpo_with_trl_ifstruct.ipynb). The results show that even a light fine-tuning procedure improves performance **from 22.6% to 29.7%** on the IFStruct benchmark.\n\nStructured output is one of the most common real-world tasks for LLMs, yet most benchmarks fold it into broader reasoning or extraction scores rather than measuring it on its own. Whether a model reliably returns valid, parseable output in the requested format and shape — schema compliance — is often what decides whether it can be wired into a downstream system at all.\n\n*Note that the training pipeline described here is not the one used to train the RL model described in the [IFStruct blog](https://www.liquid.ai/blog/ifstruct-v1.0). This notebook doesn't aim to recreate the IFStruct benchmark score, but to show how task-specific fine-tuning of smaller models can improve performance and match that of far larger models.*\n\n## Prerequisites\n\nThis guide has two halves that run in different places:\n\n- **Fine-tuning** runs on a GPU. The accompanying notebook is sized for a free-tier Colab or Kaggle GPU.\n- **Evaluation** can run locally on a MacBook (here, a MacBook Pro with an Apple M5 Max and 36 GB of unified memory) through `llama.cpp`, which exposes an OpenAI-compatible server that the IFStruct evaluator talks to.\n\nWe will need [`uv`](https://docs.astral.sh/uv/) for the Python tooling and `llama.cpp` for serving. Following the [Liquid AI llama.cpp deployment docs](https://docs.liquid.ai/deployment/on-device/llama-cpp), install `llama.cpp` with Homebrew and verify that `llama-server` is available:\n\n```shell\nbrew install llama.cpp\nllama-server --version\n```\n\n## IFStruct Evaluation on LFM2.5-350M (Base model)\n\nBefore we begin, let's evaluate LFM2.5-350M on the [IFStruct benchmark](https://huggingface.co/datasets/LiquidAI/ifstruct-v1.0) and see whether we can **reproduce the reported score of 21.1%**.\n\n**IFStruct** is a benchmark for testing the validity of LLM outputs and schema adherence. The benchmark is open-source in [Liquid4All/ifstruct](https://github.com/Liquid4All/ifstruct), with the public benchmark dataset available on Hugging Face at [LiquidAI/ifstruct-v1.0](https://huggingface.co/datasets/LiquidAI/ifstruct-v1.0).\n\n```shell\ngit clone https://github.com/Liquid4All/ifstruct.git\n```\n\nFor the eval comparison, we serve the model locally on the MacBook with `llama.cpp`. We will use the `BF16` GGUF ([LiquidAI/LFM2.5-350M-GGUF](https://huggingface.co/LiquidAI/LFM2.5-350M-GGUF)).\n\nThen we start the base-model server with the following command:\n\n```shell\nllama-server \\\n -hf LiquidAI/LFM2.5-350M-GGUF:BF16 \\\n -c 32768 \\\n -np 4 \\\n -ngl 99 \\\n --alias LiquidAI/LFM2.5-350M \\\n --host 127.0.0.1 \\\n --port 8080\n```\n\n- `--alias`: model name IFStruct sends to the OpenAI-compatible endpoint\n- `-ngl 99`: asks `llama.cpp` to offload all layers to the GPU when available\n- `-np 4`: serves four requests in parallel\n- `-c 32768`: size of the prompt context\n\nOnce the server is running, we can run the full benchmark with 2000 samples:\n\n```shell\nuv run ifstruct-eval \\\n --model LiquidAI/LFM2.5-350M \\\n --base-url http://localhost:8080/v1 \\\n --api-key dummy \\\n --dataset data/test.jsonl \\\n --results-file results/lfm2.5-350m-llamacpp-base.json \\\n --n-threads 4 \\\n --max-tokens 2048 \\\n -v\n```\n\n```\n============================================================\nModel: LiquidAI/LFM2.5-350M\n============================================================\nOverall: 452/2000 passed (22.6%)\nAverage latency: 1453ms\n\nBy format:\n JSON: 180/1000 passed (18.0%)\n YAML: 272/1000 passed (27.2%)\n\nBy top-level structure:\n Wrapper key 288/1011 passed (28.5%)\n Bare list 164/989 passed (16.6%)\n\nBy entity type:\n test__camera_review 6/83 passed (7.2%)\n test__clinical_trial 20/104 passed (19.2%)\n test__conference_schedule 7/87 passed (8.0%)\n test__escaping__bug_report_batch 24/89 passed (27.0%)\n test__escaping__config_snippet_audit 15/85 passed (17.6%)\n test__escaping__customer_email_thread 5/73 passed (6.8%)\n test__escaping__dialogue_sample 14/95 passed (14.7%)\n test__escaping__interview_transcript_segment 21/80 passed (26.2%)\n test__escaping__log_parser_examples 21/72 passed (29.2%)\n test__escaping__pr_discussion 22/87 passed (25.3%)\n test__escaping__repro_steps_batch 16/73 passed (21.9%)\n test__escaping__screenplay_scene 16/92 passed (17.4%)\n test__escaping__short_story_chapter 15/84 passed (17.9%)\n test__escaping__support_ticket_batch 27/73 passed (37.0%)\n test__escaping__terminal_session_notes 20/70 passed (28.6%)\n test__event_ticket_booking 49/107 passed (45.8%)\n test__gpu_review 6/94 passed (6.4%)\n test__invoice 28/86 passed (32.6%)\n test__job_posting 25/85 passed (29.4%)\n test__real_estate_listing 31/82 passed (37.8%)\n test__recipe 3/70 passed (4.3%)\n test__rental_car_booking 27/79 passed (34.2%)\n test__scientific_experiment 13/69 passed (18.8%)\n test__travel_itinerary 21/81 passed (25.9%)\n\nCommon errors:\n 7228x required field missing\n 738x wrong item count\n 540x type mismatch\n 317x Unclosed code block\n 190x extraneous field 'notes'\n 181x extraneous field 'path'\n 175x extraneous field 'constraints'\n 170x extraneous field 'type'\n 170x missing code block\n 100x expected bare list, got wrapper\n```\n\nThe [IFStruct release blog reports 21.1% for LFM2.5-350M](https://www.liquid.ai/blog/ifstruct-v1.0). Our local llama.cpp/BF16 setup measures 22.6%, close to the 21.1% reported in the IFStruct blog. We use this local result as the baseline for the same serving stack comparison.\n\n## GRPO Fine-tuning with TRL on Structured Outputs\n\nThe full, runnable pipeline lives in the [accompanying notebook](https://github.com/Liquid4All/cookbook/blob/main/finetuning/notebooks/grpo_with_trl_ifstruct.ipynb). We will cover only the relevant pieces in this section.\n\n### Training data\n\nWe use [`nvidia/Nemotron-RL-instruction_following-structured_outputs`](https://huggingface.co/datasets/nvidia/Nemotron-RL-instruction_following-structured_outputs), which pairs each prompt with a target JSON Schema and an expected field count. We use about 500 samples for training.\n\nBecause the Nemotron data distribution differs from the IFStruct evaluation, we augment the prompts to close two gaps between them:\n\n- **40%** get a \"return the output inside a fenced code block\" instruction appended, so the model learns to *follow* the format instruction rather than always emitting raw JSON.\n- A disjoint **20%** are converted into top-level-array tasks (the schema is wrapped in an `array` with a required item count), which trains bare-list output and item-count compliance.\n\n### Model and LoRA\n\nWe load `LiquidAI/LFM2.5-350M` and attach a LoRA adapter. Because LFM2.5 uses a hybrid attention/convolution architecture, we target the LFM-specific module names:\n\n```py\nlora_config = LoraConfig(\n r=16, \n lora_alpha=32, \n bias=\"none\", \n task_type=\"CAUSAL_LM\",\n target_modules=[\n \"q_proj\", \"k_proj\", \"v_proj\", \"out_proj\", \"in_proj\",\n \"w1\", \"w2\", \"w3\",\n ],\n)\n```\n\nThis trains ~6M parameters, about 1.66% of the model.\n\n### Reward functions\n\nThen we define three reward functions, each on a `[0, 1]` scale, which score every completion on whether the extracted *structure* is correct:\n\n- `json_format_reward`: Is the output parseable, and in the requested form? Full credit (`1.0`) for the requested form (fenced vs. raw), `0.2` for the wrong-but-parseable form, `0.0` for unparseable output.\n- `field_count_reward`: Does the object have the expected number of top-level fields? An exact match earns `1.0`, and the score decays linearly with the miss.\n- `schema_validation_reward`: Does the output validate against the row's JSON Schema? It counts every constraint violation and gates partial credit on required-key coverage.\n\nWe combine the three as a weighted sum with `reward_weights=[1.0, 0.5, 2.0]`.\n\n### Training\n\nWe train for 100 steps with 8 generations per prompt group, sized for a free-tier 16 GB GPU:\n\n```py\nfrom trl import GRPOConfig\n\ntraining_args = GRPOConfig(\n output_dir=\"./outputs/lfm25-350m-nemotron-schema-grpo\",\n learning_rate=5e-5,\n max_steps=100,\n warmup_steps=10,\n num_generations=8, # completions sampled per prompt group\n per_device_train_batch_size=4,\n gradient_accumulation_steps=8, # 4 prompt groups per optimizer step\n steps_per_generation=2,\n max_completion_length=1024, # room for nested JSON\n mask_truncated_completions=False,\n temperature=1.1, # hotter sampling keeps groups varied\n beta=0.01, # KL penalty toward the reference model\n reward_weights=[1.0, 0.5, 2.0], # json_format, field_count, schema_validation\n logging_steps=1,\n save_steps=100,\n)\n```\n\nAs you can see in the notebook, over the run, all three reward components climb, the KL from the reference model lifts off zero after warmup, and the truncated-completion fraction stays near zero.\n\n### Merging and saving the model\n\nFinally, we merge the LoRA adapter back into the base weights and save it as a single self-contained checkpoint, ready to convert to GGUF for serving:\n\n```py\nMERGED_DIR = f\"{training_args.output_dir}-merged\"\n\nmerged_model = trainer.model.merge_and_unload()\nmerged_model.save_pretrained(MERGED_DIR)\ntokenizer.save_pretrained(MERGED_DIR)\n```\n\n## IFStruct Evaluation on GRPO Tuned LFM2.5-350M\n\nAfter GRPO fine-tuning, we rerun the IFStruct evaluation. For this, we need to convert the merged model checkpoint into a BF16 GGUF. The converter script ships with the llama.cpp source, so we clone the repo once and install the converter's `gguf` package.\n\n```shell\ngit clone --depth 1 https://github.com/ggml-org/llama.cpp\npip install ./llama.cpp/gguf-py\n\nmkdir -p models\npython llama.cpp/convert_hf_to_gguf.py \\\n PATH_TO_YOUR_MERGED_MODEL \\\n --outfile ./models/lfm25-350m-grpo-bf16.gguf \\\n --outtype bf16\n```\n\nThen we serve the merged model with the following command:\n\n```shell\nllama-server \\\n -m ./models/lfm25-350m-grpo-bf16.gguf \\\n --alias lfm25-350m-grpo-structured-output \\\n -c 32768 \\\n -np 4 \\\n -ngl 99 \\\n --host 127.0.0.1 \\\n --port 8081\n```\n\nThen, we will run the full IFStruct evaluation again with the fine-tuned model:\n\n```shell\nuv run ifstruct-eval \\\n --model lfm25-350m-grpo-structured-output \\\n --base-url http://localhost:8081/v1 \\\n --api-key dummy \\\n --dataset data/test.jsonl \\\n --results-file results/lfm25-350m-grpo.json \\\n --n-threads 4 \\\n --max-tokens 2048 \\\n -v\n```\n\n```\n============================================================\nModel: lfm25-350m-grpo-structured-output\n============================================================\nOverall: 594/2000 passed (29.7%)\nAverage latency: 1518ms\n\nBy format:\n JSON: 319/1000 passed (31.9%)\n YAML: 275/1000 passed (27.5%)\n\nBy top-level structure:\n Wrapper key 300/1011 passed (29.7%)\n Bare list 294/989 passed (29.7%)\n\nBy entity type:\n test__camera_review 5/83 passed (6.0%)\n test__clinical_trial 31/104 passed (29.8%)\n test__conference_schedule 11/87 passed (12.6%)\n test__escaping__bug_report_batch 32/89 passed (36.0%)\n test__escaping__config_snippet_audit 24/85 passed (28.2%)\n test__escaping__customer_email_thread 9/73 passed (12.3%)\n test__escaping__dialogue_sample 17/95 passed (17.9%)\n test__escaping__interview_transcript_segment 13/80 passed (16.2%)\n test__escaping__log_parser_examples 33/72 passed (45.8%)\n test__escaping__pr_discussion 26/87 passed (29.9%)\n test__escaping__repro_steps_batch 23/73 passed (31.5%)\n test__escaping__screenplay_scene 34/92 passed (37.0%)\n test__escaping__short_story_chapter 24/84 passed (28.6%)\n test__escaping__support_ticket_batch 36/73 passed (49.3%)\n test__escaping__terminal_session_notes 23/70 passed (32.9%)\n test__event_ticket_booking 62/107 passed (57.9%)\n test__gpu_review 7/94 passed (7.4%)\n test__invoice 36/86 passed (41.9%)\n test__job_posting 33/85 passed (38.8%)\n test__real_estate_listing 32/82 passed (39.0%)\n test__recipe 7/70 passed (10.0%)\n test__rental_car_booking 37/79 passed (46.8%)\n test__scientific_experiment 14/69 passed (20.3%)\n test__travel_itinerary 25/81 passed (30.9%)\n\nCommon errors:\n 7331x required field missing\n 890x wrong item count\n 555x type mismatch\n 102x expected bare list, got wrapper\n 62x extraneous field 'metadata.tone'\n 55x 6 is greater than maximum 5\n 49x extraneous field 'speaker_labels'\n 47x extraneous field 'tone'\n 44x 'cups' not in allowed values ['mg', 'g', 'kg', 'oz', 'lb', 'ml', 'l', 'cl', 'dl'\n 44x extraneous field 'notes'\n```\n\nComparing the two runs on the identical serving stack:\n\n| IFStruct group | base | GRPO-tuned | Δ |\n| -------------- | ----- | ---------- | -------- |\n| **Overall** | 22.6% | **29.7%** | **+7.1** |\n| JSON | 18.0% | 31.9% | +13.9 |\n| YAML | 27.2% | 27.5% | +0.3 |\n| Wrapper key | 28.5% | 29.7% | +1.2 |\n| Bare list | 16.6% | 29.7% | +13.1 |\n\nThe gains land exactly where the training aimed: the JSON pass rate rises by nearly 14 points (18.0% → 31.9%), while YAML stays mostly the same. While this is still below the [Qwen3.5-2B score of 33.15%](https://www.liquid.ai/blog/ifstruct-v1.0), it shows that even light task-specific fine-tuning can bring a small model close to a larger one.\n\n## Conclusion\n\nA short GRPO run with about 500 samples and 100 steps can lift a small 350M parameter model from 22.6% to 29.7% on IFStruct. The takeaway is that a cheap, task-specific reward signal can make a small model substantially more reliable about *form*, closing much of the gap to models several times its size.\n\nTo reproduce or extend this work, see the original [IFStruct v1.0 blog post](https://www.liquid.ai/blog/ifstruct-v1.0), the [Liquid4All/ifstruct](https://github.com/Liquid4All/ifstruct) benchmark repo, and the [LiquidAI/ifstruct-v1.0](https://huggingface.co/datasets/LiquidAI/ifstruct-v1.0) dataset.\n", "authors": [], "captured_at": "2026-10-19T01:26:53.998049939+00:00", "category": "ai", "content_html": "

This guide is a fully public, inexpensive recipe for making a small model substantially better at structured-output compliance. We fine-tune LFM2.5-350M with Group Relative Policy Optimization (GRPO) using the TRL library and evaluate it on the IFStruct benchmark. The full run takes around 500 samples and 100 training steps, small enough for a free-tier Colab or Kaggle GPU, and is available on GitHub. The results show that even a light fine-tuning procedure improves performance from 22.6% to 29.7% on the IFStruct benchmark.

\n

Structured output is one of the most common real-world tasks for LLMs, yet most benchmarks fold it into broader reasoning or extraction scores rather than measuring it on its own. Whether a model reliably returns valid, parseable output in the requested format and shape — schema compliance — is often what decides whether it can be wired into a downstream system at all.

\n

Note that the training pipeline described here is not the one used to train the RL model described in the IFStruct blog. This notebook doesn't aim to recreate the IFStruct benchmark score, but to show how task-specific fine-tuning of smaller models can improve performance and match that of far larger models.

\n

Prerequisites

\n

This guide has two halves that run in different places:

\n\n

We will need uv for the Python tooling and llama.cpp for serving. Following the Liquid AI llama.cpp deployment docs, install llama.cpp with Homebrew and verify that llama-server is available:

\n
brew install llama.cpp\nllama-server --version\n
\n

IFStruct Evaluation on LFM2.5-350M (Base model)

\n

Before we begin, let's evaluate LFM2.5-350M on the IFStruct benchmark and see whether we can reproduce the reported score of 21.1%.

\n

IFStruct is a benchmark for testing the validity of LLM outputs and schema adherence. The benchmark is open-source in Liquid4All/ifstruct, with the public benchmark dataset available on Hugging Face at LiquidAI/ifstruct-v1.0.

\n
git clone https://github.com/Liquid4All/ifstruct.git\n
\n

For the eval comparison, we serve the model locally on the MacBook with llama.cpp. We will use the BF16 GGUF (LiquidAI/LFM2.5-350M-GGUF).

\n

Then we start the base-model server with the following command:

\n
llama-server \\\n  -hf LiquidAI/LFM2.5-350M-GGUF:BF16 \\\n  -c 32768 \\\n  -np 4 \\\n  -ngl 99 \\\n  --alias LiquidAI/LFM2.5-350M \\\n  --host 127.0.0.1 \\\n  --port 8080\n
\n\n

Once the server is running, we can run the full benchmark with 2000 samples:

\n
uv run ifstruct-eval \\\n  --model LiquidAI/LFM2.5-350M \\\n  --base-url http://localhost:8080/v1 \\\n  --api-key dummy \\\n  --dataset data/test.jsonl \\\n  --results-file results/lfm2.5-350m-llamacpp-base.json \\\n  --n-threads 4 \\\n  --max-tokens 2048 \\\n  -v\n
\n
============================================================\nModel: LiquidAI/LFM2.5-350M\n============================================================\nOverall: 452/2000 passed (22.6%)\nAverage latency: 1453ms\n\nBy format:\n  JSON: 180/1000 passed (18.0%)\n  YAML: 272/1000 passed (27.2%)\n\nBy top-level structure:\n  Wrapper key 288/1011 passed (28.5%)\n  Bare list   164/989 passed (16.6%)\n\nBy entity type:\n  test__camera_review                 6/83 passed (7.2%)\n  test__clinical_trial                20/104 passed (19.2%)\n  test__conference_schedule           7/87 passed (8.0%)\n  test__escaping__bug_report_batch    24/89 passed (27.0%)\n  test__escaping__config_snippet_audit 15/85 passed (17.6%)\n  test__escaping__customer_email_thread 5/73 passed (6.8%)\n  test__escaping__dialogue_sample     14/95 passed (14.7%)\n  test__escaping__interview_transcript_segment 21/80 passed (26.2%)\n  test__escaping__log_parser_examples 21/72 passed (29.2%)\n  test__escaping__pr_discussion       22/87 passed (25.3%)\n  test__escaping__repro_steps_batch   16/73 passed (21.9%)\n  test__escaping__screenplay_scene    16/92 passed (17.4%)\n  test__escaping__short_story_chapter 15/84 passed (17.9%)\n  test__escaping__support_ticket_batch 27/73 passed (37.0%)\n  test__escaping__terminal_session_notes 20/70 passed (28.6%)\n  test__event_ticket_booking          49/107 passed (45.8%)\n  test__gpu_review                    6/94 passed (6.4%)\n  test__invoice                       28/86 passed (32.6%)\n  test__job_posting                   25/85 passed (29.4%)\n  test__real_estate_listing           31/82 passed (37.8%)\n  test__recipe                        3/70 passed (4.3%)\n  test__rental_car_booking            27/79 passed (34.2%)\n  test__scientific_experiment         13/69 passed (18.8%)\n  test__travel_itinerary              21/81 passed (25.9%)\n\nCommon errors:\n  7228x required field missing\n  738x wrong item count\n  540x type mismatch\n  317x Unclosed code block\n  190x extraneous field 'notes'\n  181x extraneous field 'path'\n  175x extraneous field 'constraints'\n  170x extraneous field 'type'\n  170x missing code block\n  100x expected bare list, got wrapper\n
\n

The IFStruct release blog reports 21.1% for LFM2.5-350M. Our local llama.cpp/BF16 setup measures 22.6%, close to the 21.1% reported in the IFStruct blog. We use this local result as the baseline for the same serving stack comparison.

\n

GRPO Fine-tuning with TRL on Structured Outputs

\n

The full, runnable pipeline lives in the accompanying notebook. We will cover only the relevant pieces in this section.

\n

Training data

\n

We use nvidia/Nemotron-RL-instruction_following-structured_outputs, which pairs each prompt with a target JSON Schema and an expected field count. We use about 500 samples for training.

\n

Because the Nemotron data distribution differs from the IFStruct evaluation, we augment the prompts to close two gaps between them:

\n\n

Model and LoRA

\n

We load LiquidAI/LFM2.5-350M and attach a LoRA adapter. Because LFM2.5 uses a hybrid attention/convolution architecture, we target the LFM-specific module names:

\n
lora_config = LoraConfig(\n    r=16, \n    lora_alpha=32, \n    bias="none", \n    task_type="CAUSAL_LM",\n    target_modules=[\n        "q_proj", "k_proj", "v_proj", "out_proj", "in_proj",\n        "w1", "w2", "w3",\n    ],\n)\n
\n

This trains ~6M parameters, about 1.66% of the model.

\n

Reward functions

\n

Then we define three reward functions, each on a [0, 1] scale, which score every completion on whether the extracted structure is correct:

\n\n

We combine the three as a weighted sum with reward_weights=[1.0, 0.5, 2.0].

\n

Training

\n

We train for 100 steps with 8 generations per prompt group, sized for a free-tier 16 GB GPU:

\n
from trl import GRPOConfig\n\ntraining_args = GRPOConfig(\n    output_dir="./outputs/lfm25-350m-nemotron-schema-grpo",\n    learning_rate=5e-5,\n    max_steps=100,\n    warmup_steps=10,\n    num_generations=8,              # completions sampled per prompt group\n    per_device_train_batch_size=4,\n    gradient_accumulation_steps=8,  # 4 prompt groups per optimizer step\n    steps_per_generation=2,\n    max_completion_length=1024,     # room for nested JSON\n    mask_truncated_completions=False,\n    temperature=1.1,                # hotter sampling keeps groups varied\n    beta=0.01,                      # KL penalty toward the reference model\n    reward_weights=[1.0, 0.5, 2.0], # json_format, field_count, schema_validation\n    logging_steps=1,\n    save_steps=100,\n)\n
\n

As you can see in the notebook, over the run, all three reward components climb, the KL from the reference model lifts off zero after warmup, and the truncated-completion fraction stays near zero.

\n

Merging and saving the model

\n

Finally, we merge the LoRA adapter back into the base weights and save it as a single self-contained checkpoint, ready to convert to GGUF for serving:

\n
MERGED_DIR = f"{training_args.output_dir}-merged"\n\nmerged_model = trainer.model.merge_and_unload()\nmerged_model.save_pretrained(MERGED_DIR)\ntokenizer.save_pretrained(MERGED_DIR)\n
\n

IFStruct Evaluation on GRPO Tuned LFM2.5-350M

\n

After GRPO fine-tuning, we rerun the IFStruct evaluation. For this, we need to convert the merged model checkpoint into a BF16 GGUF. The converter script ships with the llama.cpp source, so we clone the repo once and install the converter's gguf package.

\n
git clone --depth 1 https://github.com/ggml-org/llama.cpp\npip install ./llama.cpp/gguf-py\n\nmkdir -p models\npython llama.cpp/convert_hf_to_gguf.py \\\n  PATH_TO_YOUR_MERGED_MODEL \\\n  --outfile ./models/lfm25-350m-grpo-bf16.gguf \\\n  --outtype bf16\n
\n

Then we serve the merged model with the following command:

\n
llama-server \\\n  -m ./models/lfm25-350m-grpo-bf16.gguf \\\n  --alias lfm25-350m-grpo-structured-output \\\n  -c 32768 \\\n  -np 4 \\\n  -ngl 99 \\\n  --host 127.0.0.1 \\\n  --port 8081\n
\n

Then, we will run the full IFStruct evaluation again with the fine-tuned model:

\n
uv run ifstruct-eval \\\n  --model lfm25-350m-grpo-structured-output \\\n  --base-url http://localhost:8081/v1 \\\n  --api-key dummy \\\n  --dataset data/test.jsonl \\\n  --results-file results/lfm25-350m-grpo.json \\\n  --n-threads 4 \\\n  --max-tokens 2048 \\\n  -v\n
\n
============================================================\nModel: lfm25-350m-grpo-structured-output\n============================================================\nOverall: 594/2000 passed (29.7%)\nAverage latency: 1518ms\n\nBy format:\n  JSON: 319/1000 passed (31.9%)\n  YAML: 275/1000 passed (27.5%)\n\nBy top-level structure:\n  Wrapper key 300/1011 passed (29.7%)\n  Bare list   294/989 passed (29.7%)\n\nBy entity type:\n  test__camera_review                 5/83 passed (6.0%)\n  test__clinical_trial                31/104 passed (29.8%)\n  test__conference_schedule           11/87 passed (12.6%)\n  test__escaping__bug_report_batch    32/89 passed (36.0%)\n  test__escaping__config_snippet_audit 24/85 passed (28.2%)\n  test__escaping__customer_email_thread 9/73 passed (12.3%)\n  test__escaping__dialogue_sample     17/95 passed (17.9%)\n  test__escaping__interview_transcript_segment 13/80 passed (16.2%)\n  test__escaping__log_parser_examples 33/72 passed (45.8%)\n  test__escaping__pr_discussion       26/87 passed (29.9%)\n  test__escaping__repro_steps_batch   23/73 passed (31.5%)\n  test__escaping__screenplay_scene    34/92 passed (37.0%)\n  test__escaping__short_story_chapter 24/84 passed (28.6%)\n  test__escaping__support_ticket_batch 36/73 passed (49.3%)\n  test__escaping__terminal_session_notes 23/70 passed (32.9%)\n  test__event_ticket_booking          62/107 passed (57.9%)\n  test__gpu_review                    7/94 passed (7.4%)\n  test__invoice                       36/86 passed (41.9%)\n  test__job_posting                   33/85 passed (38.8%)\n  test__real_estate_listing           32/82 passed (39.0%)\n  test__recipe                        7/70 passed (10.0%)\n  test__rental_car_booking            37/79 passed (46.8%)\n  test__scientific_experiment         14/69 passed (20.3%)\n  test__travel_itinerary              25/81 passed (30.9%)\n\nCommon errors:\n  7331x required field missing\n  890x wrong item count\n  555x type mismatch\n  102x expected bare list, got wrapper\n   62x extraneous field 'metadata.tone'\n   55x 6 is greater than maximum 5\n   49x extraneous field 'speaker_labels'\n   47x extraneous field 'tone'\n   44x 'cups' not in allowed values ['mg', 'g', 'kg', 'oz', 'lb', 'ml', 'l', 'cl', 'dl'\n   44x extraneous field 'notes'\n
\n

Comparing the two runs on the identical serving stack:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
IFStruct groupbaseGRPO-tunedΔ
Overall22.6%29.7%+7.1
JSON18.0%31.9%+13.9
YAML27.2%27.5%+0.3
Wrapper key28.5%29.7%+1.2
Bare list16.6%29.7%+13.1
\n

The gains land exactly where the training aimed: the JSON pass rate rises by nearly 14 points (18.0% → 31.9%), while YAML stays mostly the same. While this is still below the Qwen3.5-2B score of 33.15%, it shows that even light task-specific fine-tuning can bring a small model close to a larger one.

\n

Conclusion

\n

A short GRPO run with about 500 samples and 100 steps can lift a small 350M parameter model from 22.6% to 29.7% on IFStruct. The takeaway is that a cheap, task-specific reward signal can make a small model substantially more reliable about form, closing much of the gap to models several times its size.

\n

To reproduce or extend this work, see the original IFStruct v1.0 blog post, the Liquid4All/ifstruct benchmark repo, and the LiquidAI/ifstruct-v1.0 dataset.

\n", "content_markdown": "This guide is a fully public, inexpensive recipe for making a small model substantially better at structured-output compliance. We fine-tune [LFM2.5-350M](https://huggingface.co/LiquidAI/LFM2.5-350M) with Group Relative Policy Optimization (GRPO) using the [TRL library](https://huggingface.co/docs/trl/en/index) and evaluate it on the [IFStruct benchmark](https://huggingface.co/datasets/LiquidAI/ifstruct-v1.0). The full run takes around 500 samples and 100 training steps, small enough for a free-tier Colab or Kaggle GPU, and is available on [GitHub](https://github.com/Liquid4All/cookbook/blob/main/finetuning/notebooks/grpo_with_trl_ifstruct.ipynb). The results show that even a light fine-tuning procedure improves performance **from 22.6% to 29.7%** on the IFStruct benchmark.\n\nStructured output is one of the most common real-world tasks for LLMs, yet most benchmarks fold it into broader reasoning or extraction scores rather than measuring it on its own. Whether a model reliably returns valid, parseable output in the requested format and shape — schema compliance — is often what decides whether it can be wired into a downstream system at all.\n\n*Note that the training pipeline described here is not the one used to train the RL model described in the [IFStruct blog](https://www.liquid.ai/blog/ifstruct-v1.0). This notebook doesn't aim to recreate the IFStruct benchmark score, but to show how task-specific fine-tuning of smaller models can improve performance and match that of far larger models.*\n\n## Prerequisites\n\nThis guide has two halves that run in different places:\n\n- **Fine-tuning** runs on a GPU. The accompanying notebook is sized for a free-tier Colab or Kaggle GPU.\n- **Evaluation** can run locally on a MacBook (here, a MacBook Pro with an Apple M5 Max and 36 GB of unified memory) through `llama.cpp`, which exposes an OpenAI-compatible server that the IFStruct evaluator talks to.\n\nWe will need [`uv`](https://docs.astral.sh/uv/) for the Python tooling and `llama.cpp` for serving. Following the [Liquid AI llama.cpp deployment docs](https://docs.liquid.ai/deployment/on-device/llama-cpp), install `llama.cpp` with Homebrew and verify that `llama-server` is available:\n\n```shell\nbrew install llama.cpp\nllama-server --version\n```\n\n## IFStruct Evaluation on LFM2.5-350M (Base model)\n\nBefore we begin, let's evaluate LFM2.5-350M on the [IFStruct benchmark](https://huggingface.co/datasets/LiquidAI/ifstruct-v1.0) and see whether we can **reproduce the reported score of 21.1%**.\n\n**IFStruct** is a benchmark for testing the validity of LLM outputs and schema adherence. The benchmark is open-source in [Liquid4All/ifstruct](https://github.com/Liquid4All/ifstruct), with the public benchmark dataset available on Hugging Face at [LiquidAI/ifstruct-v1.0](https://huggingface.co/datasets/LiquidAI/ifstruct-v1.0).\n\n```shell\ngit clone https://github.com/Liquid4All/ifstruct.git\n```\n\nFor the eval comparison, we serve the model locally on the MacBook with `llama.cpp`. We will use the `BF16` GGUF ([LiquidAI/LFM2.5-350M-GGUF](https://huggingface.co/LiquidAI/LFM2.5-350M-GGUF)).\n\nThen we start the base-model server with the following command:\n\n```shell\nllama-server \\\n -hf LiquidAI/LFM2.5-350M-GGUF:BF16 \\\n -c 32768 \\\n -np 4 \\\n -ngl 99 \\\n --alias LiquidAI/LFM2.5-350M \\\n --host 127.0.0.1 \\\n --port 8080\n```\n\n- `--alias`: model name IFStruct sends to the OpenAI-compatible endpoint\n- `-ngl 99`: asks `llama.cpp` to offload all layers to the GPU when available\n- `-np 4`: serves four requests in parallel\n- `-c 32768`: size of the prompt context\n\nOnce the server is running, we can run the full benchmark with 2000 samples:\n\n```shell\nuv run ifstruct-eval \\\n --model LiquidAI/LFM2.5-350M \\\n --base-url http://localhost:8080/v1 \\\n --api-key dummy \\\n --dataset data/test.jsonl \\\n --results-file results/lfm2.5-350m-llamacpp-base.json \\\n --n-threads 4 \\\n --max-tokens 2048 \\\n -v\n```\n\n```\n============================================================\nModel: LiquidAI/LFM2.5-350M\n============================================================\nOverall: 452/2000 passed (22.6%)\nAverage latency: 1453ms\n\nBy format:\n JSON: 180/1000 passed (18.0%)\n YAML: 272/1000 passed (27.2%)\n\nBy top-level structure:\n Wrapper key 288/1011 passed (28.5%)\n Bare list 164/989 passed (16.6%)\n\nBy entity type:\n test__camera_review 6/83 passed (7.2%)\n test__clinical_trial 20/104 passed (19.2%)\n test__conference_schedule 7/87 passed (8.0%)\n test__escaping__bug_report_batch 24/89 passed (27.0%)\n test__escaping__config_snippet_audit 15/85 passed (17.6%)\n test__escaping__customer_email_thread 5/73 passed (6.8%)\n test__escaping__dialogue_sample 14/95 passed (14.7%)\n test__escaping__interview_transcript_segment 21/80 passed (26.2%)\n test__escaping__log_parser_examples 21/72 passed (29.2%)\n test__escaping__pr_discussion 22/87 passed (25.3%)\n test__escaping__repro_steps_batch 16/73 passed (21.9%)\n test__escaping__screenplay_scene 16/92 passed (17.4%)\n test__escaping__short_story_chapter 15/84 passed (17.9%)\n test__escaping__support_ticket_batch 27/73 passed (37.0%)\n test__escaping__terminal_session_notes 20/70 passed (28.6%)\n test__event_ticket_booking 49/107 passed (45.8%)\n test__gpu_review 6/94 passed (6.4%)\n test__invoice 28/86 passed (32.6%)\n test__job_posting 25/85 passed (29.4%)\n test__real_estate_listing 31/82 passed (37.8%)\n test__recipe 3/70 passed (4.3%)\n test__rental_car_booking 27/79 passed (34.2%)\n test__scientific_experiment 13/69 passed (18.8%)\n test__travel_itinerary 21/81 passed (25.9%)\n\nCommon errors:\n 7228x required field missing\n 738x wrong item count\n 540x type mismatch\n 317x Unclosed code block\n 190x extraneous field 'notes'\n 181x extraneous field 'path'\n 175x extraneous field 'constraints'\n 170x extraneous field 'type'\n 170x missing code block\n 100x expected bare list, got wrapper\n```\n\nThe [IFStruct release blog reports 21.1% for LFM2.5-350M](https://www.liquid.ai/blog/ifstruct-v1.0). Our local llama.cpp/BF16 setup measures 22.6%, close to the 21.1% reported in the IFStruct blog. We use this local result as the baseline for the same serving stack comparison.\n\n## GRPO Fine-tuning with TRL on Structured Outputs\n\nThe full, runnable pipeline lives in the [accompanying notebook](https://github.com/Liquid4All/cookbook/blob/main/finetuning/notebooks/grpo_with_trl_ifstruct.ipynb). We will cover only the relevant pieces in this section.\n\n### Training data\n\nWe use [`nvidia/Nemotron-RL-instruction_following-structured_outputs`](https://huggingface.co/datasets/nvidia/Nemotron-RL-instruction_following-structured_outputs), which pairs each prompt with a target JSON Schema and an expected field count. We use about 500 samples for training.\n\nBecause the Nemotron data distribution differs from the IFStruct evaluation, we augment the prompts to close two gaps between them:\n\n- **40%** get a \"return the output inside a fenced code block\" instruction appended, so the model learns to *follow* the format instruction rather than always emitting raw JSON.\n- A disjoint **20%** are converted into top-level-array tasks (the schema is wrapped in an `array` with a required item count), which trains bare-list output and item-count compliance.\n\n### Model and LoRA\n\nWe load `LiquidAI/LFM2.5-350M` and attach a LoRA adapter. Because LFM2.5 uses a hybrid attention/convolution architecture, we target the LFM-specific module names:\n\n```py\nlora_config = LoraConfig(\n r=16, \n lora_alpha=32, \n bias=\"none\", \n task_type=\"CAUSAL_LM\",\n target_modules=[\n \"q_proj\", \"k_proj\", \"v_proj\", \"out_proj\", \"in_proj\",\n \"w1\", \"w2\", \"w3\",\n ],\n)\n```\n\nThis trains ~6M parameters, about 1.66% of the model.\n\n### Reward functions\n\nThen we define three reward functions, each on a `[0, 1]` scale, which score every completion on whether the extracted *structure* is correct:\n\n- `json_format_reward`: Is the output parseable, and in the requested form? Full credit (`1.0`) for the requested form (fenced vs. raw), `0.2` for the wrong-but-parseable form, `0.0` for unparseable output.\n- `field_count_reward`: Does the object have the expected number of top-level fields? An exact match earns `1.0`, and the score decays linearly with the miss.\n- `schema_validation_reward`: Does the output validate against the row's JSON Schema? It counts every constraint violation and gates partial credit on required-key coverage.\n\nWe combine the three as a weighted sum with `reward_weights=[1.0, 0.5, 2.0]`.\n\n### Training\n\nWe train for 100 steps with 8 generations per prompt group, sized for a free-tier 16 GB GPU:\n\n```py\nfrom trl import GRPOConfig\n\ntraining_args = GRPOConfig(\n output_dir=\"./outputs/lfm25-350m-nemotron-schema-grpo\",\n learning_rate=5e-5,\n max_steps=100,\n warmup_steps=10,\n num_generations=8, # completions sampled per prompt group\n per_device_train_batch_size=4,\n gradient_accumulation_steps=8, # 4 prompt groups per optimizer step\n steps_per_generation=2,\n max_completion_length=1024, # room for nested JSON\n mask_truncated_completions=False,\n temperature=1.1, # hotter sampling keeps groups varied\n beta=0.01, # KL penalty toward the reference model\n reward_weights=[1.0, 0.5, 2.0], # json_format, field_count, schema_validation\n logging_steps=1,\n save_steps=100,\n)\n```\n\nAs you can see in the notebook, over the run, all three reward components climb, the KL from the reference model lifts off zero after warmup, and the truncated-completion fraction stays near zero.\n\n### Merging and saving the model\n\nFinally, we merge the LoRA adapter back into the base weights and save it as a single self-contained checkpoint, ready to convert to GGUF for serving:\n\n```py\nMERGED_DIR = f\"{training_args.output_dir}-merged\"\n\nmerged_model = trainer.model.merge_and_unload()\nmerged_model.save_pretrained(MERGED_DIR)\ntokenizer.save_pretrained(MERGED_DIR)\n```\n\n## IFStruct Evaluation on GRPO Tuned LFM2.5-350M\n\nAfter GRPO fine-tuning, we rerun the IFStruct evaluation. For this, we need to convert the merged model checkpoint into a BF16 GGUF. The converter script ships with the llama.cpp source, so we clone the repo once and install the converter's `gguf` package.\n\n```shell\ngit clone --depth 1 https://github.com/ggml-org/llama.cpp\npip install ./llama.cpp/gguf-py\n\nmkdir -p models\npython llama.cpp/convert_hf_to_gguf.py \\\n PATH_TO_YOUR_MERGED_MODEL \\\n --outfile ./models/lfm25-350m-grpo-bf16.gguf \\\n --outtype bf16\n```\n\nThen we serve the merged model with the following command:\n\n```shell\nllama-server \\\n -m ./models/lfm25-350m-grpo-bf16.gguf \\\n --alias lfm25-350m-grpo-structured-output \\\n -c 32768 \\\n -np 4 \\\n -ngl 99 \\\n --host 127.0.0.1 \\\n --port 8081\n```\n\nThen, we will run the full IFStruct evaluation again with the fine-tuned model:\n\n```shell\nuv run ifstruct-eval \\\n --model lfm25-350m-grpo-structured-output \\\n --base-url http://localhost:8081/v1 \\\n --api-key dummy \\\n --dataset data/test.jsonl \\\n --results-file results/lfm25-350m-grpo.json \\\n --n-threads 4 \\\n --max-tokens 2048 \\\n -v\n```\n\n```\n============================================================\nModel: lfm25-350m-grpo-structured-output\n============================================================\nOverall: 594/2000 passed (29.7%)\nAverage latency: 1518ms\n\nBy format:\n JSON: 319/1000 passed (31.9%)\n YAML: 275/1000 passed (27.5%)\n\nBy top-level structure:\n Wrapper key 300/1011 passed (29.7%)\n Bare list 294/989 passed (29.7%)\n\nBy entity type:\n test__camera_review 5/83 passed (6.0%)\n test__clinical_trial 31/104 passed (29.8%)\n test__conference_schedule 11/87 passed (12.6%)\n test__escaping__bug_report_batch 32/89 passed (36.0%)\n test__escaping__config_snippet_audit 24/85 passed (28.2%)\n test__escaping__customer_email_thread 9/73 passed (12.3%)\n test__escaping__dialogue_sample 17/95 passed (17.9%)\n test__escaping__interview_transcript_segment 13/80 passed (16.2%)\n test__escaping__log_parser_examples 33/72 passed (45.8%)\n test__escaping__pr_discussion 26/87 passed (29.9%)\n test__escaping__repro_steps_batch 23/73 passed (31.5%)\n test__escaping__screenplay_scene 34/92 passed (37.0%)\n test__escaping__short_story_chapter 24/84 passed (28.6%)\n test__escaping__support_ticket_batch 36/73 passed (49.3%)\n test__escaping__terminal_session_notes 23/70 passed (32.9%)\n test__event_ticket_booking 62/107 passed (57.9%)\n test__gpu_review 7/94 passed (7.4%)\n test__invoice 36/86 passed (41.9%)\n test__job_posting 33/85 passed (38.8%)\n test__real_estate_listing 32/82 passed (39.0%)\n test__recipe 7/70 passed (10.0%)\n test__rental_car_booking 37/79 passed (46.8%)\n test__scientific_experiment 14/69 passed (20.3%)\n test__travel_itinerary 25/81 passed (30.9%)\n\nCommon errors:\n 7331x required field missing\n 890x wrong item count\n 555x type mismatch\n 102x expected bare list, got wrapper\n 62x extraneous field 'metadata.tone'\n 55x 6 is greater than maximum 5\n 49x extraneous field 'speaker_labels'\n 47x extraneous field 'tone'\n 44x 'cups' not in allowed values ['mg', 'g', 'kg', 'oz', 'lb', 'ml', 'l', 'cl', 'dl'\n 44x extraneous field 'notes'\n```\n\nComparing the two runs on the identical serving stack:\n\n| IFStruct group | base | GRPO-tuned | Δ |\n| -------------- | ----- | ---------- | -------- |\n| **Overall** | 22.6% | **29.7%** | **+7.1** |\n| JSON | 18.0% | 31.9% | +13.9 |\n| YAML | 27.2% | 27.5% | +0.3 |\n| Wrapper key | 28.5% | 29.7% | +1.2 |\n| Bare list | 16.6% | 29.7% | +13.1 |\n\nThe gains land exactly where the training aimed: the JSON pass rate rises by nearly 14 points (18.0% → 31.9%), while YAML stays mostly the same. While this is still below the [Qwen3.5-2B score of 33.15%](https://www.liquid.ai/blog/ifstruct-v1.0), it shows that even light task-specific fine-tuning can bring a small model close to a larger one.\n\n## Conclusion\n\nA short GRPO run with about 500 samples and 100 steps can lift a small 350M parameter model from 22.6% to 29.7% on IFStruct. The takeaway is that a cheap, task-specific reward signal can make a small model substantially more reliable about *form*, closing much of the gap to models several times its size.\n\nTo reproduce or extend this work, see the original [IFStruct v1.0 blog post](https://www.liquid.ai/blog/ifstruct-v1.0), the [Liquid4All/ifstruct](https://github.com/Liquid4All/ifstruct) benchmark repo, and the [LiquidAI/ifstruct-v1.0](https://huggingface.co/datasets/LiquidAI/ifstruct-v1.0) dataset.\n", "dateCreated": "2026-10-19T01:26:53.998049939+00:00", "date_created": "2026-09-03T00:00:00+00:00", "date_modified": "2026-09-03T00:00:00+00:00", "date_published": "2026-09-03T00:00:00+00:00", "description": "This guide is a fully public, inexpensive recipe for making a small model substantially better at structured-output compliance. We fine-tune LFM2.5-350M with Group Relative Policy Optimization (GRPO) using the TRL library and evaluate it…", "external_url": "https://huggingface.co/blog/grpo-with-trl-ifstruct", "git": { "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e", "path": "items/huggingface-co/2026/09/2026-09-03-fine-tuning-a-350m-model-for-better-structured-outputs-in.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/huggingface-co/2026/09/2026-09-03-fine-tuning-a-350m-model-for-better-structured-outputs-in.md" }, "id": "urn:aggr:item:f2b3a710f6484274c8de002544d238d2af41e78c", "isBasedOn": { "@id": "https://huggingface.co/blog/grpo-with-trl-ifstruct" }, "mainEntity": { "@id": "https://huggingface.co/blog/grpo-with-trl-ifstruct", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-09-03-fine-tuning-a-350m-model-for-better-structured-outputs-in/#webpage" }, "dateModified": "2026-09-03T00:00:00+00:00", "datePublished": "2026-09-03T00:00:00+00:00", "headline": "Fine-tuning a 350M Model for Better Structured Outputs in 100 GRPO Steps", "image": { "@type": "ImageObject", "caption": null, "height": 90, "url": "https://huggingface.co/blog/assets/grpo-with-trl-ifstruct/thumbnail.png", "width": 160 }, "timeRequired": "PT8M", "url": "https://huggingface.co/blog/grpo-with-trl-ifstruct", "wordCount": 1796 }, "name": "Archived snapshot: Fine-tuning a 350M Model for Better Structured Outputs in 100 GRPO Steps", "preview": { "alt": null, "color": null, "height": 90, "placeholder": { "data_url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII=", "hash": "" }, "url": "https://huggingface.co/blog/assets/grpo-with-trl-ifstruct/thumbnail.png", "width": 160 }, "reading_minutes": 8, "replicated_at": null, "source": "huggingface-co", "source_name": "Hugging Face - Blog", "summary": null, "tags": [], "temporalCoverage": "2026-10-19T01:26:53.998049939+00:00", "title": "Fine-tuning a 350M Model for Better Structured Outputs in 100 GRPO Steps", "url": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-09-03-fine-tuning-a-350m-model-for-better-structured-outputs-in/", "word_count": 1796 }