{
  "@context": "https://schema.org",
  "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-08-25-granite-4-2-llms-how-they-re-built/#webpage",
  "@type": [
    "WebPage",
    "ArchiveComponent"
  ],
  "articleBody": "*A technical walkthrough of how we built the Granite 4.2 reasoning model family.*\n\n**Authors:** Granite Team, IBM\n\n* * *\n\n**TL;DR:** Granite 4.2 is our first family of dense, decoder-only reasoning LLMs, released in three sizes: **3B, 8B, and 30B**. These models are post-trained from [Granite-4.1](https://huggingface.co/collections/ibm-granite/granite-41-language-models) base models. [Granite-4.1](https://huggingface.co/collections/ibm-granite/granite-41-language-models) base models were pre-trained from scratch on roughly 15T tokens with a five-phase strategy that extends the context window to 512K tokens, supervised fine-tuned on chain-of-thought, reasoning, and agentic-trajectory data, then post-trained with a **multi-stage reinforcement learning pipeline**. That pipeline includes agentic RL, where the 8B and 30B models learn to act with tools inside real sandboxed environments. Every model has a **thinking / non-thinking** switch, a **low-effort** thinking mode that spends a short reasoning budget on easy questions, and native tool calling. All Granite 4.2 models are released under the Apache 2.0 license.\n\n**Links:**\n\n- [Granite 4.2 HF Collection](https://huggingface.co/collections/ibm-granite/granite-42-language-models)\n- [GitHub Repository](https://github.com/ibm-granite/granite-4.2-language-models)\n- [Granite Docs](https://www.ibm.com/granite/docs/)\n\n* * *\n\n## Overview\n\nGranite 4.2 is the reasoning-focused release of the Granite language-model family. Earlier Granite releases were strong instruction-following assistants; Granite 4.2 adds explicit reasoning. Every model can produce a chain of thought before its answer and can run in **thinking** or **non-thinking** mode depending on how much deliberation a task needs. A **low-effort** mode falls between the two, spending a short reasoning budget on easy questions.\n\nThe three sizes (**3B, 8B, and 30B**) share the same architectural design and follow the same training pipeline (base model, SFT, then multi-stage RL), each at its own scale. All three are strong reasoners and instruction followers. The clearest capability split shows up in post-training. The **8B and 30B** models additionally go through an **agentic RL** block that teaches them to operate as agents: calling tools, editing and running code, driving a terminal, and searching the web inside real environments. Every model supports native tool calling. Served through an OpenAI-compatible endpoint (for example, with vLLM), it emits tool calls in the OpenAI function-calling format and plugs into agentic harnesses without extra glue. Granite 4.2 is also supported in SGLang, see the [SGLang cookbook](https://docs.sglang.io/cookbook/autoregressive/IBM/Granite-4.2) for a ready-to-serve recipe.\n\nThe rest of this post walks through the build: architecture, pre-training, supervised fine-tuning, the multi-stage RL pipeline, and results.\n\n* * *\n\n## Model Architecture\n\nGranite 4.2 models are built on a decoder-only dense transformer architecture with the following core components:\n\n- **Attention:** Grouped Query Attention (GQA) with 40 attention heads and 8 KV heads\n- **Position Embedding:** Rotary Position Embedding (RoPE) with θ = 10,000,000\n- **Feed-Forward:** MLP with SwiGLU activation\n- **Normalization:** RMSNorm (ε = 1e-5)\n- **Embeddings:** Separate input/output embeddings (not tied)\n- **Precision:** bfloat16\n\n| Component                 | 3B Dense | 8B Dense | 30B Dense |\n| ------------------------- | -------- | -------- | --------- |\n| Embedding size            | 2560     | 4096     | 4096      |\n| Number of layers          | 40       | 40       | 64        |\n| Attention head size       | 64       | 128      | 128       |\n| Number of attention heads | 40       | 32       | 32        |\n| Number of KV heads        | 8        | 8        | 8         |\n| MLP hidden size           | 8192     | 12800    | 32768     |\n| MLP activation            | SwiGLU   | SwiGLU   | SwiGLU    |\n| Sequence length           | 131072   | 131072   | 131072    |\n| Position embedding        | RoPE     | RoPE     | RoPE      |\n| \\# Parameters             | 3B       | 8B       | 30B       |\n\n* * *\n\n## Pre-Training\n\nGranite 4.2 models are post-trained from [Granite-4.1](https://huggingface.co/collections/ibm-granite/granite-41-language-models) base models. [Granite-4.1](https://huggingface.co/collections/ibm-granite/granite-41-language-models) base models were trained from scratch on approximately **15 trillion tokens** using a five-phase training strategy. Phases 1–2 focus on foundational pre-training, phases 3–4 perform mid-training with progressively higher-quality data annealing, and phase 5 introduces long-context training, extending the context window to **512K tokens**. Each phase uses a distinct data mixture and learning-rate schedule, gradually shifting from broad web-scale data toward more curated, high-quality sources.\n\nThe pre-training recipe closely follows the previous generation; for a detailed treatment of the data blend, phase schedule, and long-context extension, see the [Granite 4.1 blog](https://huggingface.co/blog/ibm-granite/granite-4-1).\n\n* * *\n\n## SFT: Data Preparation & Quality Control\n\nSupervised fine-tuning (SFT) turns the base model into a reliable instruction-following, reasoning, and tool-using assistant. The SFT data mixture combines agentic (31.6%) and non-agentic (68.4%) data, totaling approximately 7.2 million samples, or roughly 100B tokens, of which about 65B are trainable.\n\nThe **agentic corpus** covers a broad range of domains, including software engineering (SWE, 69%), tool calling (12.1%), terminal use (8.0%), math (3.5%), search (0.8%), and action (0.2%). These samples and trajectories are generated using a diverse set of agent scaffolds and harnesses, including OpenHands, OpenCode, Terminus-2, SWE-agent, OpenResearcher, MiniSWE, OpenSeeker, EnvScaler, Gemini CLI, Hermes, Codex, and Goose. The agentic data combines samples from both open-source datasets and our own synthetically generated RL environments, spanning a variety of agent–harness combinations.\n\nThe **non-agentic corpus** consists of several major categories: instruction following (18.8%), coding (18.8%), math (14.6%), multilingual (7.0%), science (5.4%), reasoning (3.0%), and safety (0.8%).\n\n### Data Quality Control\n\nWe apply multiple stages of quality control before a sample enters the final SFT mixture. First, data from different sources is normalized and reformatted into a consistent OpenAI Chat format, making the conversation structure and tool interactions uniform across datasets and scaffolds.\n\nWe then use GPT-OSS-120B and Gemma 4 as LLM-based judges to assess sample quality. Low-scoring samples are removed, as are samples containing hallucinated or fabricated information, invalid tool interactions, or tool calls to functions that are not defined in the corresponding tool list. Several targeted, dataset-specific heuristic rules are also applied where appropriate to further improve quality and remove known sources of noise.\n\nFinally, we perform both local and global deduplication. Deduplication is based on SHA-256 hashes computed over the combination of the `tools` and `messages` fields, removing duplicate samples both within individual data sources and across the overall SFT mixture.\n\n### SFT Training Details\n\nThe complete corpus is first globally shuffled to reduce ordering effects and ensure that samples from different domains are well mixed during training. The shuffled corpus is then partitioned into equally sized `.parquet` shards, which are tokenized using the model's tokenizer and chat template and prepared for large-scale distributed training.\n\nBefore launching the final large-scale runs, we tune hyperparameters on representative configurations, sweeping learning-rate schedules, initial learning rates, and warm-up ratios to find settings that train stably across model sizes. The final training configuration is summarized below:\n\n| Parameter                | Value                                                 |\n| ------------------------ | ----------------------------------------------------- |\n| Compute                  | 32–128 nodes (by model size), 4× Grace/GB200 per node |\n| Sequence length (packed) | 131,072 (128K)                                        |\n| Global batch size        | 128                                                   |\n| Learning rate            | 1.0e-5, constant after warm-up; 3.0e-6 for Phase 2    |\n| LR warm-up               | 2.5% of `train_iters`                                 |\n| Training duration        | \\~2 epochs                                            |\n| Parallelism              | TP=2, PP=1, CP=4 or CP=2                              |\n\n### Phase 2 SFT for the 30B Model\n\nFor the 30B model, we additionally perform a second phase of SFT focused specifically on agentic coding. In this phase, agentic, SWE, and coding data are upsampled to increase their effective contribution to the training distribution, while approximately 16% of the mixture is retained as replay data from the original SFT corpus.\n\nThe 30B model is then fine-tuned for roughly one additional epoch at a lower learning rate of 3.0e-6. This targeted second phase increases the model's exposure to agentic coding trajectories without discarding the capabilities acquired during the initial SFT phase.\n\n* * *\n\n## Reinforcement Learning: A Multi-Stage, Multi-Environment Pipeline\n\nAfter SFT, we apply a **multi-stage, multi-environment reinforcement learning pipeline**. Rather than a single RL pass, we run a *chain* of focused stages spanning many environments: math, code, science, instruction following, tool use, and structured output, then software engineering, terminal use, and web search. Each stage is an independent RL run that targets one capability and warm-starts from the previous stage's checkpoint.\n\n[![Granite 4.2 staged RL curriculum](https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/TuxEn5hma0bQf3-z_Ab6c.png)](https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/TuxEn5hma0bQf3-z_Ab6c.png)\n\n*Figure 1. The staged RL curriculum. Foundational RL (verifiable rewards + skill boosters) runs for all sizes; the agentic RL block (SWE → Terminal → Search) runs for 8B and 30B only. Every model finishes with RLHF. Each stage is a separate GRPO run that warm-starts from the previous checkpoint.*\n\n### Training Methodology\n\nEvery stage trains with **asynchronous GRPO** ([Group Relative Policy Optimization](https://arxiv.org/abs/2402.03300)), so the generator and trainer halves of the loop never block on each other. A pool of generation workers keeps sampling responses and dropping the finished trajectories into a shared buffer; once the buffer holds a full step's worth, the trainer pulls that batch, takes an optimizer step, and streams the updated parameters back to the generator workers without pausing them. A refresh can land partway through a rollout, leaving a single trajectory stitched together from two adjacent policy versions. We allow this instead of paying to prevent it: the workers reuse their existing KV cache rather than rebuilding it after each refresh, and the one guardrail is a limit that keeps them from drifting more than a single update behind the trainer, which bounds how off-policy any sample can get. Whatever mismatch survives that limit is handled in the objective by *truncated importance sampling*, which clamps the train-versus-generation log-probability ratio to a fixed ceiling so a handful of stale tokens cannot dominate an update.\n\nAdvantages are group-relative with a **leave-one-out baseline**: each response is judged against the mean reward of the *other* samples drawn for the same prompt, which removes the need for a separate value network. To make this concrete, take **RLVR**, the first and longest-running stage: each step pairs **256 prompts** with **16 sampled responses apiece** for a **4,096**-example batch, which the trainer consumes in a single optimizer step before the next rollout begins. Later stages keep this machinery unchanged and adjust only the per-stage shape, shown next.\n\n#### RL training configuration\n\nThe pipeline keeps a common backbone of hyperparameters across every stage, which makes the curriculum easier to run and compare. A handful of knobs are fixed everywhere:\n\n| Parameter              | Value (shared across stages)                              |\n| ---------------------- | --------------------------------------------------------- |\n| Algorithm              | GRPO (no value network; group-relative advantages)        |\n| Training stack         | NeMo-RL (Megatron-Core + vLLM) with NeMo-Gym environments |\n| Ratio clip (min / max) | 0.2 / 0.28                                                |\n| Micro-batch size       | 1                                                         |\n| Parallelism            | tensor-parallel 2–4; no pipeline- or context-parallelism  |\n\nWhat changes from stage to stage is the *shape* of each run: how many prompts and generations per step, how long the context is, whether the agent loop runs, and how hard we pull back toward the reference policy. The table below gives the exact settings for the **30B** chain, stage by stage:\n\n| Stage        | Prompts/step | Gens/prompt | Max seq len | Rollout turns | KL   | LR       |\n| ------------ | ------------ | ----------- | ----------- | ------------- | ---- | -------- |\n| RLVR (×3)    | 256          | 16          | 64K         | 1             | 0    | 5e-7     |\n| IF booster   | 256          | 16          | 64K         | 1             | 0    | 5e-7     |\n| Code booster | 64           | 16          | 64K         | 1             | 0.05 | 5e-7     |\n| SWE 1        | 64           | 16          | 128K        | 1             | 0.01 | 5e-7     |\n| SWE 2        | 32           | 16          | 128K        | **128**       | 0    | 5e-7     |\n| Terminal     | 8            | 32          | 64K         | **64**        | 0.01 | **1e-6** |\n| Search       | 32           | 16          | 128K        | **64**        | 0.01 | 5e-7     |\n| RLHF         | 128          | 16          | 48K         | 1             | 0.05 | 5e-7     |\n\n*Parameters shown for the 30B model. Global batch size = prompts/step × generations/prompt (e.g. 256 × 16 = 4096 for RLVR). The 3B and 8B models use the same recipe and hyperparameters with **fewer stages** (see [How the Three Sizes Differ](https://huggingface.co/blog/ibm-granite/granite-4-2#how-the-three-sizes-differ)); the stage list is what changes, not the knobs.*\n\nThe **KL schedule** follows the reward type: explore freely where the reward is objective and verifiable (RLVR and SWE 2 run at KL 0), and stay close to the reference where the objective is preference, safety, or a narrow skill graft (RLHF and the code booster use KL 0.05). The **rollout-turns** column counts the environment interactions *GRPO itself* sees per rollout. In every case the model is trained on complete, real-environment trajectories.\n\n### The Staged Curriculum\n\nEach stage is a separate RL run with a single objective and its own reward signal. When it finishes, its policy is exported to Hugging Face format and becomes the base model for the next stage, so the pipeline is a sequence of warm-starts:\n\n```\nSFT ─▶ RLVR ─▶ Skill boosters ─▶ SWE agent ─▶ Terminal ─▶ Search ─▶ RLHF\n      └──────── foundational RL ────────┘   └──────── agentic RL (8B / 30B) ────────┘\n```\n\nThe **8B and 30B** models follow the full ladder. The **3B** model takes a shortened path: foundational RL and alignment, without the agentic block.\n\n### Reward Signals\n\nA stage is defined mostly by *how it is rewarded*. Across the pipeline there are three reward types, and a single stage can use more than one:\n\n| Reward type                  | What it measures                                                              | Used in                 |\n| ---------------------------- | ----------------------------------------------------------------------------- | ----------------------- |\n| **Verifiable**               | Exact-match, unit tests, format checkers, rule-based checkers on ground truth | RLVR · boosters · SWE   |\n| **Reward model / LLM judge** | Open-ended quality, preference, safety, answer correctness                    | RLVR · Search · RLHF    |\n| **Agentic outcome**          | Did the model actually solve a task in a real environment?                    | SWE · Terminal · Search |\n\nVerifiable rewards are objective and hard to game, so the pipeline front-loads them. Judge- and preference-based rewards handle open-ended qualities that no checker can express. Agentic-outcome rewards are the sparsest: often a single bit at the end of a long tool-use trajectory.\n\n### Foundational RL: Build the Skills\n\n#### RLVR: verifiable-reward RL\n\nRLVR is the foundational stage and the broadest data mix in the pipeline: a single blended dataset spanning many verifiable domains.\n\n- **Math:** chain-of-thought with boxed-answer checking, plus formal proving in **Lean**\n- **Competitive coding:** solutions checked against hidden tests in a sandbox\n- **STEM / graduate-level science MCQA** and general knowledge\n- **Instruction following:** structured-output and inverse-instruction tasks\n- **Tool / function calling:** single-step tool use\n- **Reasoning puzzles** and **abstention** (knowing when to refuse)\n\nEach task type carries its own verifier, so the reward is grounded per example. RLVR runs for **two rounds on 3B and 8B, and three on 30B**. Each round is a fresh warm-started run on a re-weighted mix of public and internally curated RL data.\n\n#### Skill boosters: targeted lifts\n\nAfter RLVR, a few short **booster** stages sharpen specific capabilities that benefit from concentrated training focusing on the following domains:\n\n- **Instruction following (IF):** multi-turn chat, inverse-IFEval, structured outputs\n- **Code:** competitive coding only\n\nBoosters are small, focused runs. A light KL penalty keeps the model close to its current behavior while nudging one skill.\n\n### Agentic RL: Learning to Act (8B / 30B)\n\nIn the agentic stages the model learns to **act**: call tools, observe results, and iterate inside a real environment, rewarded on whether the task was actually solved. These stages share the same shape: multi-turn tool use, real (not simulated) environments, sparse outcome rewards, and GRPO, warm-started from the coding-boosted checkpoint. They run in order: **SWE → Terminal → Search**.\n\n[![Agentic RL environments](https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/FxSMYasHnR4vpv263_DAL.png)](https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/FxSMYasHnR4vpv263_DAL.png)\n\n*Figure 2. The three agentic-RL environments. Each pairs a real harness with a real environment and a sparse, outcome-based reward. The 3B model runs none of these.*\n\n- **SWE agent (software engineering).** Each task is a real repository in its own sandbox. Driven by the [OpenHands](https://github.com/All-Hands-AI/OpenHands) harness, the model reads code, edits files, and runs the test suite over many internal turns. The reward is verifiable: do the hidden tests pass? Tasks are drawn from open-source SWE datasets, each instance backed by a per-repo container image.\n- **Terminal agent (terminal / OS operation).** Multi-step tasks in a live shell, run through the Harbor / [Terminus-2](https://www.harborframework.com/docs/agents/terminus-2) agent harness. The model plans a sequence of commands, observes their output, and recovers from errors. Reward is assigned when the task completed successfully. This is the one stage that drives its multi-turn agent loop at the GRPO level, with rollouts spanning up to 64 environment turns.\n- **Search agent (deep research).** The model answers hard, multi-hop questions using live web-search tool calls inside a browsing agent loop: gather evidence across hops, reason over it, and produce an answer. Because correctness here is open-ended, the reward is an LLM judge on the final answer.\n\n### Alignment: RLHF\n\nThe final stage of every model is **RLHF for human preference and safety.** It optimizes against a generative reward model (GenRM) for preference, plus a safety reward covering jailbreak resistance and appropriate refusals. This stage uses the highest KL penalty in the pipeline, aligning tone and safety without eroding the capabilities the earlier stages built. In addition to human preference and safety alignment, this stage also applies a reasoning-length penalty to discourage overly verbose reasoning behavior acquired during earlier stages.\n\n### How the Three Sizes Differ\n\nSame method and infrastructure; the difference is how far up the ladder each model goes.\n\n| Stage                          | 3B   | 8B               | 30B       |\n| ------------------------------ | ---- | ---------------- | --------- |\n| **RLVR** (verifiable)          | ×2   | ×2               | ×3        |\n| **Skill boosters**             | code | IF · GPQA · code | IF · code |\n| **SWE agent**                  | —    | ✓                | ✓         |\n| **Terminal agent**             | —    | ✓                | ✓         |\n| **Search agent**               | —    | ✓                | ✓         |\n| **RLHF** (preference + safety) | ✓    | ✓                | ✓         |\n\n3B is a strong foundational-RL model; 8B and 30B add the agentic-RL block on top, learning to act with tools in real environments.\n\n* * *\n\n## Agentic AI Infrastructure for Scalable RL\n\nReinforcement learning at this scale needs infrastructure that can drive a training loop and a fleet of live environments at the same time. This matters most in the agentic stages, where every training example is a multi-turn rollout that edits code, runs commands, or browses the web. Granite 4.2's RL runs on two open components: **[NeMo-RL](https://github.com/NVIDIA-NeMo/RL)** on the training side and **[NeMo-Gym](https://github.com/NVIDIA-NeMo/gym)** on the rollout side.\n\n[![NeMo-RL + NeMo-Gym system architecture](https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/ceRxgjK0m0kVHp1iJ7P34.png)](https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/ceRxgjK0m0kVHp1iJ7P34.png)\n\n*Figure 3. The RL system. NeMo-RL drives the GRPO loop (Megatron-Core training backend, vLLM generation, and Megatron-Bridge for HF⇄Megatron weight conversion). NeMo-Gym orchestrates rollouts and hosts the tools, sandboxes, and reward/verifier calls as pluggable **Resources**.*\n\nThe division of labor:\n\n- **NeMo-RL (training side).** [Megatron-Core](https://github.com/NVIDIA/Megatron-LM) is the training backend; [vLLM](https://github.com/vllm-project/vllm) generates rollouts; **Megatron-Bridge** converts weights between Megatron and Hugging Face formats, so each stage can export a clean HF checkpoint for the next one.\n- **NeMo-Gym (rollout side).** It exposes each environment as a set of **Resources** (verifiers, tools, sandboxes, and reward models) behind a uniform interface. This is the plug point for the agentic stages: the SWE repo sandboxes, the terminal harness, and the web-search tools all attach here, and to the training loop they look the same as a simple math verifier.\n\nThat uniformity is what makes the staged curriculum above practical: a booster's rule-based checker and a full SWE sandbox present the same interface to GRPO.\n\nThis split is also what makes the **asynchronous** training loop described above physically possible: generation and policy updates live on separate GPU pools, so the expensive generation fleet — including the live agentic environments — stays busy instead of idling through optimizer steps.\n\n* * *\n\n## Results\n\nGranite 4.2 was evaluated across agentic coding, general agentic and tool use, reasoning, chat and instruction following, and long context. The full benchmark table is below, followed by charts that break out the headline results by model size.\n\n| Task                         | 3B Dense | 8B Dense | 30B Dense |\n| ---------------------------- | -------- | -------- | --------- |\n| Agentic (Coding)             |          |          |           |\n| SWE Bench Multilingual       | NA       | 30.78    | 41.89     |\n| SWE Bench Pro                | NA       | 19.11    | 33.29     |\n| SWE Bench Verified           | NA       | 47.67    | 57.00     |\n| Terminal-Bench 2.1           | NA       | 20.56    | 29.24     |\n| Agentic (General)            |          |          |           |\n| τ³-bench                     | 45.78    | 58.06    | 62.00     |\n| BFCL (v4)                    | 52.41    | 50.29    | 61.39     |\n| ProfBench                    | 32.10    | 41.20    | 42.90     |\n| BirdBench                    | NA       | 41.07    | 41.85     |\n| GDPval                       | NA       | 1189.00  | 1225.00   |\n| Reasoning                    |          |          |           |\n| AIME25                       | 78.33    | 86.67    | 89.17     |\n| HMMT Feb25                   | 66.67    | 78.33    | 89.17     |\n| GPQA                         | 54.80    | 64.14    | 66.41     |\n| LiveCodeBench v6             | 69.71    | 73.24    | 75.77     |\n| SciCode                      | 24.11    | 36.09    | 38.76     |\n| Chat & Instruction Following |          |          |           |\n| MMLU-Pro                     | 67.84    | 74.04    | 77.60     |\n| MMLU-ProX lite (IBM)         | 27.78    | 61.06    | 66.64     |\n| Arena-Hard-V2                | 34.96    | 65.19    | 67.93     |\n| IFBench (prompt)             | 74.33    | 79.33    | 77.17     |\n| Long Context                 |          |          |           |\n| RULER 64K                    | 67.52    | 80.99    | 89.96     |\n| RULER 128K                   | 55.30    | 71.41    | 81.38     |\n\n**Supported languages:** English, German, Spanish, French, Japanese, Portuguese, Arabic, Czech, Italian, Korean, Dutch, and Chinese.\n\nThe charts below break these results out by capability area.\n\n[![Reasoning benchmarks](https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/u4p7x5-U_7Qj9w-ovsZ7D.png)](https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/u4p7x5-U_7Qj9w-ovsZ7D.png)\n\n*Figure 4. Reasoning (pass@1). Scores rise consistently with model size across math (AIME25, HMMT), science (GPQA), and code reasoning (LiveCodeBench, SciCode).*\n\n[![Agentic coding benchmarks](https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/CH5Aqnjba9hHlBteGNvw0.png)](https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/CH5Aqnjba9hHlBteGNvw0.png)\n\n*Figure 5. Agentic coding resolve rates. The agentic-RL block is trained only for 8B and 30B; the 30B model leads across SWE-Bench variants and Terminal-Bench.*\n\n[![General agentic and tool-use benchmarks](https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/mXZuw86XqEvna7Rm2OcTi.png)](https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/mXZuw86XqEvna7Rm2OcTi.png)\n\n*Figure 6. General agentic and tool-use benchmarks, reported for all three sizes.*\n\n* * *\n\n## Quantization\n\nWe also released four quantized variants of the Granite 4.2 models for inference with vLLM. The models are converted to FP8, NVFP4, and MXFP4 using LLM Compressor, and to the GGUF format using the llama.cpp framework for reduced-memory deployment.\n\n### FP8\n\nThe FP8 version is quantized with dynamic per-channel weights and per-token activations. No calibration is used.\n\n### FP4\n\nThe NVFP4 and MXFP4 versions are quantized using GPTQ calibrated on 2K samples drawn from the SFT dataset. Max context length is 2K during calibration.\n\n### GGUF\n\nConversion of the Granite 4.2 models to GGUF is done with the canonical llama.cpp tool as described in [https://github.com/IBM/gguf#gguf-conversion--quantization](https://github.com/IBM/gguf#gguf-conversion--quantization).\n\nSeveral GGUF formats are provided:\n\n- Q8\\_0\n- Q6\\_K\n- Q5\\_K\\_S\n- Q5\\_K\\_M\n- Q5\\_1\n- Q5\\_0\n- Q4\\_K\\_S\n- Q4\\_K\\_M\n- Q4\\_1\n- Q4\\_0\n- Q3\\_K\\_S\n- Q3\\_K\\_M\n- Q3\\_K\\_L\n- Q2\\_K\n\n* * *\n\n## Infrastructure\n\n### Hardware\n\nWe trained the Granite 4.2 language models on an NVIDIA GB200 NVL72 cluster hosted by CoreWeave, featuring:\n\n- A 72-GPU NVLink domain for high-speed intra-rack communication\n- A non-blocking Fat-Tree NDR 400 Gb/s InfiniBand fabric for full-bandwidth inter-rack connectivity\n- Thousands of GPUs operating at cluster scale\n\nThis infrastructure delivers the high-bandwidth, low-latency communication required for efficient large-scale distributed training.\n\n### Software Stack\n\nThe training software stack is packaged into `.sqsh` container images, each giving a run a reproducible, portable environment with its SBSA-compatible CUDA targets, Linux aarch64 Python wheels, and GPU-specific binaries pinned. The large-scale SFT runs build on an NGC PyTorch base image (Ubuntu 22.04, CUDA 12.8, Python 3.12); the RL stack runs in its own NeMo-RL container.\n\n* * *\n\n## Getting Started (Transformers)\n\n### Installation\n\n```bash\npip install torch\npip install accelerate transformers\n```\n\n### Basic Inference (Thinking Mode)\n\n```python\nimport torch\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel_path = \"ibm-granite/granite-4.2-30b\"\ntokenizer = AutoTokenizer.from_pretrained(model_path)\nmodel = AutoModelForCausalLM.from_pretrained(model_path, device_map=\"cuda\", torch_dtype=torch.bfloat16)\nmodel.eval()\n\nmessages = [\n    {\"role\": \"user\", \"content\": \"How many r's are in the word 'strawberry'?\"},\n]\n\ntext = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=True)\ninputs = tokenizer(text, return_tensors=\"pt\").to(model.device)\n\nwith torch.no_grad():\n    output = model.generate(**inputs, max_new_tokens=8192, temperature=1.0, top_p=0.95, do_sample=True)\n\nprint(tokenizer.decode(output[0][inputs.input_ids.shape[-1]:], skip_special_tokens=False))\n```\n\n**Example Output**\n\n```\n<think>\nOkay, let's see. The problem is to find how many 'r's are in the word 'strawberry'.\n\nFirst, I need to write out the word: s t r a w b e r r y.\n\nNow, I need to count the number of 'r' letters. Let's list each letter and check for 'r'.\n\n1. s – not r\n2. t – not r\n3. r – yes, that's one\n4. a – no\n5. w – no\n6. b – no\n7. e – no\n8. r – yes, that's two\n9. r – yes, that's three\n10. y – no\n\nTotal r's = 3.\n</think>\nThere are **3** r's in the word \"strawberry\".<|im_end|>\n```\n\n### Non-Thinking Mode\n\n```python\nmessages = [\n    {\"role\": \"user\", \"content\": \"What is the capital of France?\"},\n]\ntext = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)\ninputs = tokenizer(text, return_tensors=\"pt\").to(model.device)\n\noutput = model.generate(**inputs, max_new_tokens=2048, temperature=1.0, top_p=0.95, do_sample=True)\nprint(tokenizer.decode(output[0][inputs.input_ids.shape[-1]:], skip_special_tokens=False))\n```\n\n**Example Output**\n\n```\n<think></think>The capital of France is Paris.<|im_end|>\n```\n\n### Low-Effort Thinking\n\n```python\nmessages = [\n    {\"role\": \"user\", \"content\": \"What is 2 + 2?\"},\n]\ntext = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True,\n                                     enable_thinking=True, low_effort=True)\ninputs = tokenizer(text, return_tensors=\"pt\").to(model.device)\n\noutput = model.generate(**inputs, max_new_tokens=4096, temperature=1.0, top_p=0.95, do_sample=True)\nprint(tokenizer.decode(output[0][inputs.input_ids.shape[-1]:], skip_special_tokens=False))\n```\n\n**Example Output**\n\n```\n<think>\nSimple answer.\n</think>\n2 + 2 = 4.<|im_end|>\n```\n\n* * *\n\n## Tool Calling\n\nGranite models support tool calling with integrated reasoning: the model reasons about which tool to call and why before calling it. Tools are defined with the [OpenAI function definition schema](https://platform.openai.com/docs/guides/function-calling).\n\n### Basic Tool Calling\n\n```python\ntools = [\n    {\n        \"type\": \"function\",\n        \"function\": {\n            \"name\": \"get_current_weather\",\n            \"description\": \"Get the current weather for a specified city.\",\n            \"parameters\": {\n                \"type\": \"object\",\n                \"properties\": {\n                    \"city\": {\"type\": \"string\", \"description\": \"Name of the city\"}\n                },\n                \"required\": [\"city\"]\n            }\n        }\n    }\n]\n\nmessages = [\n    {\"role\": \"user\", \"content\": \"What's the weather like in Boston right now?\"},\n]\ntext = tokenizer.apply_chat_template(messages, tokenize=False, tools=tools,\n                                     add_generation_prompt=True, enable_thinking=True)\ninputs = tokenizer(text, return_tensors=\"pt\").to(model.device)\n\noutput = model.generate(**inputs, max_new_tokens=4096, temperature=1.0, top_p=0.95, do_sample=True)\nprint(tokenizer.decode(output[0][inputs.input_ids.shape[-1]:], skip_special_tokens=False))\n```\n\n**Example Output**\n\n```\n<think>\nOkay, the user is asking for the weather in Boston right now. Let me check the tools\navailable. There's a function called get_current_weather that takes a city parameter.\nI need to call that with the city set to Boston.\n</think>\n<tool_call>\n<function=get_current_weather>\n<parameter=city>\nBoston\n</parameter>\n</function>\n</tool_call>\n<|im_end|>\n```\n\n### Multi-Turn with Tool Response\n\n```python\nmessages = [\n    {\"role\": \"user\", \"content\": \"What's the weather like in Boston right now?\"},\n    {\"role\": \"assistant\", \"content\": \"<think>\\nThe user wants to know the current weather in Boston. I should call get_current_weather.\\n</think>\",\n     \"tool_calls\": [{\"function\": {\"name\": \"get_current_weather\", \"arguments\": {\"city\": \"Boston\"}}}]},\n    {\"role\": \"tool\", \"content\": '{\"temperature\": \"72°F\", \"condition\": \"Partly cloudy\", \"humidity\": \"65%\"}'},\n]\ntext = tokenizer.apply_chat_template(messages, tokenize=False, tools=tools,\n                                     add_generation_prompt=True, enable_thinking=True)\ninputs = tokenizer(text, return_tensors=\"pt\").to(model.device)\n\noutput = model.generate(**inputs, max_new_tokens=4096, temperature=1.0, top_p=0.95, do_sample=True)\nprint(tokenizer.decode(output[0][inputs.input_ids.shape[-1]:], skip_special_tokens=False))\n```\n\n**Example Output**\n\n```\n<think>\nThe tool returned the weather data for Boston: temperature 72°F, partly cloudy, humidity 65%.\nI need to present this information clearly to the user.\n</think>\nThe current weather in Boston is 72°F, partly cloudy, with 65% humidity.<|im_end|>\n```\n\n* * *\n\n## Multi-Turn Conversations\n\n### History Thinking Truncation\n\n```python\nmessages = [\n    {\"role\": \"user\", \"content\": \"What is 15 * 37?\"},\n    {\"role\": \"assistant\", \"content\": \"<think>\\nLet me calculate 15 * 37.\\n15 * 37 = 15 * 30 + 15 * 7 = 450 + 105 = 555\\n</think>\\n15 * 37 = 555\"},\n    {\"role\": \"user\", \"content\": \"Now divide that by 5\"},\n]\n\n# Default: previous thinking is stripped to save context\ntext = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True,\n                                     enable_thinking=True, truncate_history_thinking=True)\n\n# To preserve full history:\ntext_full = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True,\n                                          enable_thinking=True, truncate_history_thinking=False)\n```\n\n### Parsing Thinking vs. Final Answer\n\n```python\nimport re\n\ndef parse_model_output(text):\n    \"\"\"Separate thinking content from final answer.\"\"\"\n    think_match = re.search(r'<think>(.*?)</think>', text, re.DOTALL)\n    if think_match:\n        thinking = think_match.group(1).strip()\n        answer_start = text.find('</think>') + len('</think>')\n        answer_end = text.find('<|im_end|>', answer_start)\n        answer = text[answer_start:answer_end].strip() if answer_end != -1 else text[answer_start:].strip()\n    else:\n        thinking, answer = \"\", text.strip()\n    return thinking, answer\n\nthinking, answer = parse_model_output(output_text)\n```\n\n* * *\n\n## Using with Agentic Coding Harnesses\n\nGranite models can serve as the backbone for agentic coding tools. Because they support reasoning and tool calling through the OpenAI-compatible API, they integrate with popular agentic harnesses without extra adapters. Start the vLLM server, then follow the harness-specific instructions below.\n\n### OpenCode\n\n[OpenCode](https://opencode.ai) is an AI coding agent that runs in your terminal.\n\n**Install:**\n\n```bash\ncurl -fsSL https://opencode.ai/install | bash\n```\n\n**Configure** `~/.config/opencode/opencode.json`:\n\n```json\n{\n  \"$schema\": \"https://opencode.ai/config.json\",\n  \"model\": \"local/granite-4.2-30b\",\n  \"provider\": {\n    \"local\": {\n      \"npm\": \"@ai-sdk/openai-compatible\",\n      \"name\": \"vLLM (local)\",\n      \"options\": {\n        \"baseURL\": \"http://localhost:8000/v1\",\n        \"apiKey\": \"EMPTY\"\n      },\n      \"models\": {\n        \"granite-4.2-30b\": {\n          \"name\": \"Granite 4.2 30B\",\n          \"limit\": {\n            \"context\": 131072,\n            \"output\": 8192\n          }\n        }\n      }\n    }\n  }\n}\n```\n\n**Run:**\n\n```bash\nopencode\nopencode run \"your task description\"\n```\n\nFor full documentation, see [opencode.ai/docs](https://opencode.ai/docs).\n\n### Pi\n\n[Pi](https://pi.dev) is a minimal agent harness for AI-powered coding that runs in your terminal. It supports custom providers via a `models.json` configuration file.\n\n**Install:**\n\n```bash\ncurl -fsSL https://pi.dev/install.sh | sh\n```\n\n**Configure** `~/.pi/agent/models.json`:\n\n```json\n{\n  \"providers\": {\n    \"vllm\": {\n      \"baseUrl\": \"http://localhost:8000/v1\",\n      \"api\": \"openai-completions\",\n      \"apiKey\": \"EMPTY\",\n      \"compat\": {\n        \"supportsDeveloperRole\": false,\n        \"supportsReasoningEffort\": false\n      },\n      \"models\": [\n        {\n          \"id\": \"granite-4.2-30b\",\n          \"name\": \"Granite 4.2 30B\",\n          \"reasoning\": true,\n          \"input\": [\"text\"],\n          \"contextWindow\": 131072,\n          \"maxTokens\": 8192,\n          \"samplingParams\": {\n            \"temperature\": 1.0,\n            \"top_p\": 0.95\n          },\n          \"cost\": { \"input\": 0, \"output\": 0, \"cacheRead\": 0, \"cacheWrite\": 0 }\n        }\n      ]\n    }\n  }\n}\n```\n\n**Run:**\n\n```bash\npi\n```\n\nThen select the `granite-4.2-30b` model with `/model` or `Ctrl+L` in the interactive session.\n\nFor full documentation, see [pi.dev/docs](https://pi.dev/docs/latest).\n\n### OpenHands\n\n[OpenHands](https://www.openhands.dev) is an AI software engineer that can plan, write code, and execute commands.\n\n1. **Install and launch OpenHands** following the [official installation guide](https://docs.openhands.dev/openhands/usage/agent-canvas/setup).\n\n2. **Configure the LLM** in the OpenHands settings with:\n\n   - **Model:** `granite-4.2-30b`\n   - **Base URL:** `http://localhost:8000/v1`\n   - **API Key:** your vLLM `--api-key` value\n\n> **Note:** The `openai/` prefix is required when connecting to OpenAI-compatible endpoints like vLLM. Refer to the [OpenHands local LLM documentation](https://docs.openhands.dev/openhands/usage/llms/local-llms) for detailed setup instructions, troubleshooting, and alternative installation methods.\n\n* * *\n\n**Resources:**\n\n- [Granite 4.2 HF Collection](https://huggingface.co/collections/ibm-granite/granite-42-language-models)\n- [GitHub: ibm-granite/granite-4.2-language-models](https://github.com/ibm-granite/granite-4.2-language-models)\n- [Granite Documentation](https://www.ibm.com/granite/docs/)\n- [Granite Community Resources](https://github.com/ibm-granite-community/)\n- [Code Alchemy](https://research.ibm.com/blog/code-alchemy-for-synthetic-code)\n",
  "authors": [],
  "captured_at": "2026-10-19T01:26:53.998049939+00:00",
  "category": "ai",
  "content_html": "<p><em>A technical walkthrough of how we built the Granite 4.2 reasoning model family.</em></p>\n<p><strong>Authors:</strong> Granite Team, IBM</p>\n<hr />\n<p><strong>TL;DR:</strong> Granite 4.2 is our first family of dense, decoder-only reasoning LLMs, released in three sizes: <strong>3B, 8B, and 30B</strong>. These models are post-trained from <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://huggingface.co/collections/ibm-granite/granite-41-language-models\">Granite-4.1</a> base models. <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://huggingface.co/collections/ibm-granite/granite-41-language-models\">Granite-4.1</a> base models were pre-trained from scratch on roughly 15T tokens with a five-phase strategy that extends the context window to 512K tokens, supervised fine-tuned on chain-of-thought, reasoning, and agentic-trajectory data, then post-trained with a <strong>multi-stage reinforcement learning pipeline</strong>. That pipeline includes agentic RL, where the 8B and 30B models learn to act with tools inside real sandboxed environments. Every model has a <strong>thinking / non-thinking</strong> switch, a <strong>low-effort</strong> thinking mode that spends a short reasoning budget on easy questions, and native tool calling. All Granite 4.2 models are released under the Apache 2.0 license.</p>\n<p><strong>Links:</strong></p>\n<ul>\n<li><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://huggingface.co/collections/ibm-granite/granite-42-language-models\">Granite 4.2 HF Collection</a></li>\n<li><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://github.com/ibm-granite/granite-4.2-language-models\">GitHub Repository</a></li>\n<li><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.ibm.com/granite/docs/\">Granite Docs</a></li>\n</ul>\n<hr />\n<h2>Overview</h2>\n<p>Granite 4.2 is the reasoning-focused release of the Granite language-model family. Earlier Granite releases were strong instruction-following assistants; Granite 4.2 adds explicit reasoning. Every model can produce a chain of thought before its answer and can run in <strong>thinking</strong> or <strong>non-thinking</strong> mode depending on how much deliberation a task needs. A <strong>low-effort</strong> mode falls between the two, spending a short reasoning budget on easy questions.</p>\n<p>The three sizes (<strong>3B, 8B, and 30B</strong>) share the same architectural design and follow the same training pipeline (base model, SFT, then multi-stage RL), each at its own scale. All three are strong reasoners and instruction followers. The clearest capability split shows up in post-training. The <strong>8B and 30B</strong> models additionally go through an <strong>agentic RL</strong> block that teaches them to operate as agents: calling tools, editing and running code, driving a terminal, and searching the web inside real environments. Every model supports native tool calling. Served through an OpenAI-compatible endpoint (for example, with vLLM), it emits tool calls in the OpenAI function-calling format and plugs into agentic harnesses without extra glue. Granite 4.2 is also supported in SGLang, see the <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://docs.sglang.io/cookbook/autoregressive/IBM/Granite-4.2\">SGLang cookbook</a> for a ready-to-serve recipe.</p>\n<p>The rest of this post walks through the build: architecture, pre-training, supervised fine-tuning, the multi-stage RL pipeline, and results.</p>\n<hr />\n<h2>Model Architecture</h2>\n<p>Granite 4.2 models are built on a decoder-only dense transformer architecture with the following core components:</p>\n<ul>\n<li><strong>Attention:</strong> Grouped Query Attention (GQA) with 40 attention heads and 8 KV heads</li>\n<li><strong>Position Embedding:</strong> Rotary Position Embedding (RoPE) with θ = 10,000,000</li>\n<li><strong>Feed-Forward:</strong> MLP with SwiGLU activation</li>\n<li><strong>Normalization:</strong> RMSNorm (ε = 1e-5)</li>\n<li><strong>Embeddings:</strong> Separate input/output embeddings (not tied)</li>\n<li><strong>Precision:</strong> bfloat16</li>\n</ul>\n<table>\n<thead>\n<tr>\n<th>Component</th>\n<th>3B Dense</th>\n<th>8B Dense</th>\n<th>30B Dense</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Embedding size</td>\n<td>2560</td>\n<td>4096</td>\n<td>4096</td>\n</tr>\n<tr>\n<td>Number of layers</td>\n<td>40</td>\n<td>40</td>\n<td>64</td>\n</tr>\n<tr>\n<td>Attention head size</td>\n<td>64</td>\n<td>128</td>\n<td>128</td>\n</tr>\n<tr>\n<td>Number of attention heads</td>\n<td>40</td>\n<td>32</td>\n<td>32</td>\n</tr>\n<tr>\n<td>Number of KV heads</td>\n<td>8</td>\n<td>8</td>\n<td>8</td>\n</tr>\n<tr>\n<td>MLP hidden size</td>\n<td>8192</td>\n<td>12800</td>\n<td>32768</td>\n</tr>\n<tr>\n<td>MLP activation</td>\n<td>SwiGLU</td>\n<td>SwiGLU</td>\n<td>SwiGLU</td>\n</tr>\n<tr>\n<td>Sequence length</td>\n<td>131072</td>\n<td>131072</td>\n<td>131072</td>\n</tr>\n<tr>\n<td>Position embedding</td>\n<td>RoPE</td>\n<td>RoPE</td>\n<td>RoPE</td>\n</tr>\n<tr>\n<td># Parameters</td>\n<td>3B</td>\n<td>8B</td>\n<td>30B</td>\n</tr>\n</tbody>\n</table>\n<hr />\n<h2>Pre-Training</h2>\n<p>Granite 4.2 models are post-trained from <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://huggingface.co/collections/ibm-granite/granite-41-language-models\">Granite-4.1</a> base models. <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://huggingface.co/collections/ibm-granite/granite-41-language-models\">Granite-4.1</a> base models were trained from scratch on approximately <strong>15 trillion tokens</strong> using a five-phase training strategy. Phases 1–2 focus on foundational pre-training, phases 3–4 perform mid-training with progressively higher-quality data annealing, and phase 5 introduces long-context training, extending the context window to <strong>512K tokens</strong>. Each phase uses a distinct data mixture and learning-rate schedule, gradually shifting from broad web-scale data toward more curated, high-quality sources.</p>\n<p>The pre-training recipe closely follows the previous generation; for a detailed treatment of the data blend, phase schedule, and long-context extension, see the <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://huggingface.co/blog/ibm-granite/granite-4-1\">Granite 4.1 blog</a>.</p>\n<hr />\n<h2>SFT: Data Preparation &amp; Quality Control</h2>\n<p>Supervised fine-tuning (SFT) turns the base model into a reliable instruction-following, reasoning, and tool-using assistant. The SFT data mixture combines agentic (31.6%) and non-agentic (68.4%) data, totaling approximately 7.2 million samples, or roughly 100B tokens, of which about 65B are trainable.</p>\n<p>The <strong>agentic corpus</strong> covers a broad range of domains, including software engineering (SWE, 69%), tool calling (12.1%), terminal use (8.0%), math (3.5%), search (0.8%), and action (0.2%). These samples and trajectories are generated using a diverse set of agent scaffolds and harnesses, including OpenHands, OpenCode, Terminus-2, SWE-agent, OpenResearcher, MiniSWE, OpenSeeker, EnvScaler, Gemini CLI, Hermes, Codex, and Goose. The agentic data combines samples from both open-source datasets and our own synthetically generated RL environments, spanning a variety of agent–harness combinations.</p>\n<p>The <strong>non-agentic corpus</strong> consists of several major categories: instruction following (18.8%), coding (18.8%), math (14.6%), multilingual (7.0%), science (5.4%), reasoning (3.0%), and safety (0.8%).</p>\n<h3>Data Quality Control</h3>\n<p>We apply multiple stages of quality control before a sample enters the final SFT mixture. First, data from different sources is normalized and reformatted into a consistent OpenAI Chat format, making the conversation structure and tool interactions uniform across datasets and scaffolds.</p>\n<p>We then use GPT-OSS-120B and Gemma 4 as LLM-based judges to assess sample quality. Low-scoring samples are removed, as are samples containing hallucinated or fabricated information, invalid tool interactions, or tool calls to functions that are not defined in the corresponding tool list. Several targeted, dataset-specific heuristic rules are also applied where appropriate to further improve quality and remove known sources of noise.</p>\n<p>Finally, we perform both local and global deduplication. Deduplication is based on SHA-256 hashes computed over the combination of the <code>tools</code> and <code>messages</code> fields, removing duplicate samples both within individual data sources and across the overall SFT mixture.</p>\n<h3>SFT Training Details</h3>\n<p>The complete corpus is first globally shuffled to reduce ordering effects and ensure that samples from different domains are well mixed during training. The shuffled corpus is then partitioned into equally sized <code>.parquet</code> shards, which are tokenized using the model's tokenizer and chat template and prepared for large-scale distributed training.</p>\n<p>Before launching the final large-scale runs, we tune hyperparameters on representative configurations, sweeping learning-rate schedules, initial learning rates, and warm-up ratios to find settings that train stably across model sizes. The final training configuration is summarized below:</p>\n<table>\n<thead>\n<tr>\n<th>Parameter</th>\n<th>Value</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Compute</td>\n<td>32–128 nodes (by model size), 4× Grace/GB200 per node</td>\n</tr>\n<tr>\n<td>Sequence length (packed)</td>\n<td>131,072 (128K)</td>\n</tr>\n<tr>\n<td>Global batch size</td>\n<td>128</td>\n</tr>\n<tr>\n<td>Learning rate</td>\n<td>1.0e-5, constant after warm-up; 3.0e-6 for Phase 2</td>\n</tr>\n<tr>\n<td>LR warm-up</td>\n<td>2.5% of <code>train_iters</code></td>\n</tr>\n<tr>\n<td>Training duration</td>\n<td>~2 epochs</td>\n</tr>\n<tr>\n<td>Parallelism</td>\n<td>TP=2, PP=1, CP=4 or CP=2</td>\n</tr>\n</tbody>\n</table>\n<h3>Phase 2 SFT for the 30B Model</h3>\n<p>For the 30B model, we additionally perform a second phase of SFT focused specifically on agentic coding. In this phase, agentic, SWE, and coding data are upsampled to increase their effective contribution to the training distribution, while approximately 16% of the mixture is retained as replay data from the original SFT corpus.</p>\n<p>The 30B model is then fine-tuned for roughly one additional epoch at a lower learning rate of 3.0e-6. This targeted second phase increases the model's exposure to agentic coding trajectories without discarding the capabilities acquired during the initial SFT phase.</p>\n<hr />\n<h2>Reinforcement Learning: A Multi-Stage, Multi-Environment Pipeline</h2>\n<p>After SFT, we apply a <strong>multi-stage, multi-environment reinforcement learning pipeline</strong>. Rather than a single RL pass, we run a <em>chain</em> of focused stages spanning many environments: math, code, science, instruction following, tool use, and structured output, then software engineering, terminal use, and web search. Each stage is an independent RL run that targets one capability and warm-starts from the previous stage's checkpoint.</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/TuxEn5hma0bQf3-z_Ab6c.png\"><picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/TuxEn5hma0bQf3-z_Ab6c.png\" alt=\"Granite 4.2 staged RL curriculum\" loading=\"eager\" decoding=\"async\" fetchpriority=\"high\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture></a></p>\n<p><em>Figure 1. The staged RL curriculum. Foundational RL (verifiable rewards + skill boosters) runs for all sizes; the agentic RL block (SWE → Terminal → Search) runs for 8B and 30B only. Every model finishes with RLHF. Each stage is a separate GRPO run that warm-starts from the previous checkpoint.</em></p>\n<h3>Training Methodology</h3>\n<p>Every stage trains with <strong>asynchronous GRPO</strong> (<a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://arxiv.org/abs/2402.03300\">Group Relative Policy Optimization</a>), so the generator and trainer halves of the loop never block on each other. A pool of generation workers keeps sampling responses and dropping the finished trajectories into a shared buffer; once the buffer holds a full step's worth, the trainer pulls that batch, takes an optimizer step, and streams the updated parameters back to the generator workers without pausing them. A refresh can land partway through a rollout, leaving a single trajectory stitched together from two adjacent policy versions. We allow this instead of paying to prevent it: the workers reuse their existing KV cache rather than rebuilding it after each refresh, and the one guardrail is a limit that keeps them from drifting more than a single update behind the trainer, which bounds how off-policy any sample can get. Whatever mismatch survives that limit is handled in the objective by <em>truncated importance sampling</em>, which clamps the train-versus-generation log-probability ratio to a fixed ceiling so a handful of stale tokens cannot dominate an update.</p>\n<p>Advantages are group-relative with a <strong>leave-one-out baseline</strong>: each response is judged against the mean reward of the <em>other</em> samples drawn for the same prompt, which removes the need for a separate value network. To make this concrete, take <strong>RLVR</strong>, the first and longest-running stage: each step pairs <strong>256 prompts</strong> with <strong>16 sampled responses apiece</strong> for a <strong>4,096</strong>-example batch, which the trainer consumes in a single optimizer step before the next rollout begins. Later stages keep this machinery unchanged and adjust only the per-stage shape, shown next.</p>\n<h4>RL training configuration</h4>\n<p>The pipeline keeps a common backbone of hyperparameters across every stage, which makes the curriculum easier to run and compare. A handful of knobs are fixed everywhere:</p>\n<table>\n<thead>\n<tr>\n<th>Parameter</th>\n<th>Value (shared across stages)</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Algorithm</td>\n<td>GRPO (no value network; group-relative advantages)</td>\n</tr>\n<tr>\n<td>Training stack</td>\n<td>NeMo-RL (Megatron-Core + vLLM) with NeMo-Gym environments</td>\n</tr>\n<tr>\n<td>Ratio clip (min / max)</td>\n<td>0.2 / 0.28</td>\n</tr>\n<tr>\n<td>Micro-batch size</td>\n<td>1</td>\n</tr>\n<tr>\n<td>Parallelism</td>\n<td>tensor-parallel 2–4; no pipeline- or context-parallelism</td>\n</tr>\n</tbody>\n</table>\n<p>What changes from stage to stage is the <em>shape</em> of each run: how many prompts and generations per step, how long the context is, whether the agent loop runs, and how hard we pull back toward the reference policy. The table below gives the exact settings for the <strong>30B</strong> chain, stage by stage:</p>\n<table>\n<thead>\n<tr>\n<th>Stage</th>\n<th>Prompts/step</th>\n<th>Gens/prompt</th>\n<th>Max seq len</th>\n<th>Rollout turns</th>\n<th>KL</th>\n<th>LR</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>RLVR (×3)</td>\n<td>256</td>\n<td>16</td>\n<td>64K</td>\n<td>1</td>\n<td>0</td>\n<td>5e-7</td>\n</tr>\n<tr>\n<td>IF booster</td>\n<td>256</td>\n<td>16</td>\n<td>64K</td>\n<td>1</td>\n<td>0</td>\n<td>5e-7</td>\n</tr>\n<tr>\n<td>Code booster</td>\n<td>64</td>\n<td>16</td>\n<td>64K</td>\n<td>1</td>\n<td>0.05</td>\n<td>5e-7</td>\n</tr>\n<tr>\n<td>SWE 1</td>\n<td>64</td>\n<td>16</td>\n<td>128K</td>\n<td>1</td>\n<td>0.01</td>\n<td>5e-7</td>\n</tr>\n<tr>\n<td>SWE 2</td>\n<td>32</td>\n<td>16</td>\n<td>128K</td>\n<td><strong>128</strong></td>\n<td>0</td>\n<td>5e-7</td>\n</tr>\n<tr>\n<td>Terminal</td>\n<td>8</td>\n<td>32</td>\n<td>64K</td>\n<td><strong>64</strong></td>\n<td>0.01</td>\n<td><strong>1e-6</strong></td>\n</tr>\n<tr>\n<td>Search</td>\n<td>32</td>\n<td>16</td>\n<td>128K</td>\n<td><strong>64</strong></td>\n<td>0.01</td>\n<td>5e-7</td>\n</tr>\n<tr>\n<td>RLHF</td>\n<td>128</td>\n<td>16</td>\n<td>48K</td>\n<td>1</td>\n<td>0.05</td>\n<td>5e-7</td>\n</tr>\n</tbody>\n</table>\n<p><em>Parameters shown for the 30B model. Global batch size = prompts/step × generations/prompt (e.g. 256 × 16 = 4096 for RLVR). The 3B and 8B models use the same recipe and hyperparameters with <strong>fewer stages</strong> (see <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://huggingface.co/blog/ibm-granite/granite-4-2#how-the-three-sizes-differ\">How the Three Sizes Differ</a>); the stage list is what changes, not the knobs.</em></p>\n<p>The <strong>KL schedule</strong> follows the reward type: explore freely where the reward is objective and verifiable (RLVR and SWE 2 run at KL 0), and stay close to the reference where the objective is preference, safety, or a narrow skill graft (RLHF and the code booster use KL 0.05). The <strong>rollout-turns</strong> column counts the environment interactions <em>GRPO itself</em> sees per rollout. In every case the model is trained on complete, real-environment trajectories.</p>\n<h3>The Staged Curriculum</h3>\n<p>Each stage is a separate RL run with a single objective and its own reward signal. When it finishes, its policy is exported to Hugging Face format and becomes the base model for the next stage, so the pipeline is a sequence of warm-starts:</p>\n<pre><code><span class=\"code-snippet\">SFT ─▶ RLVR ─▶ Skill boosters ─▶ SWE agent ─▶ Terminal ─▶ Search ─▶ RLHF\n      └──────── foundational RL ────────┘   └──────── agentic RL (8B / 30B) ────────┘\n</span></code></pre>\n<p>The <strong>8B and 30B</strong> models follow the full ladder. The <strong>3B</strong> model takes a shortened path: foundational RL and alignment, without the agentic block.</p>\n<h3>Reward Signals</h3>\n<p>A stage is defined mostly by <em>how it is rewarded</em>. Across the pipeline there are three reward types, and a single stage can use more than one:</p>\n<table>\n<thead>\n<tr>\n<th>Reward type</th>\n<th>What it measures</th>\n<th>Used in</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td><strong>Verifiable</strong></td>\n<td>Exact-match, unit tests, format checkers, rule-based checkers on ground truth</td>\n<td>RLVR · boosters · SWE</td>\n</tr>\n<tr>\n<td><strong>Reward model / LLM judge</strong></td>\n<td>Open-ended quality, preference, safety, answer correctness</td>\n<td>RLVR · Search · RLHF</td>\n</tr>\n<tr>\n<td><strong>Agentic outcome</strong></td>\n<td>Did the model actually solve a task in a real environment?</td>\n<td>SWE · Terminal · Search</td>\n</tr>\n</tbody>\n</table>\n<p>Verifiable rewards are objective and hard to game, so the pipeline front-loads them. Judge- and preference-based rewards handle open-ended qualities that no checker can express. Agentic-outcome rewards are the sparsest: often a single bit at the end of a long tool-use trajectory.</p>\n<h3>Foundational RL: Build the Skills</h3>\n<h4>RLVR: verifiable-reward RL</h4>\n<p>RLVR is the foundational stage and the broadest data mix in the pipeline: a single blended dataset spanning many verifiable domains.</p>\n<ul>\n<li><strong>Math:</strong> chain-of-thought with boxed-answer checking, plus formal proving in <strong>Lean</strong></li>\n<li><strong>Competitive coding:</strong> solutions checked against hidden tests in a sandbox</li>\n<li><strong>STEM / graduate-level science MCQA</strong> and general knowledge</li>\n<li><strong>Instruction following:</strong> structured-output and inverse-instruction tasks</li>\n<li><strong>Tool / function calling:</strong> single-step tool use</li>\n<li><strong>Reasoning puzzles</strong> and <strong>abstention</strong> (knowing when to refuse)</li>\n</ul>\n<p>Each task type carries its own verifier, so the reward is grounded per example. RLVR runs for <strong>two rounds on 3B and 8B, and three on 30B</strong>. Each round is a fresh warm-started run on a re-weighted mix of public and internally curated RL data.</p>\n<h4>Skill boosters: targeted lifts</h4>\n<p>After RLVR, a few short <strong>booster</strong> stages sharpen specific capabilities that benefit from concentrated training focusing on the following domains:</p>\n<ul>\n<li><strong>Instruction following (IF):</strong> multi-turn chat, inverse-IFEval, structured outputs</li>\n<li><strong>Code:</strong> competitive coding only</li>\n</ul>\n<p>Boosters are small, focused runs. A light KL penalty keeps the model close to its current behavior while nudging one skill.</p>\n<h3>Agentic RL: Learning to Act (8B / 30B)</h3>\n<p>In the agentic stages the model learns to <strong>act</strong>: call tools, observe results, and iterate inside a real environment, rewarded on whether the task was actually solved. These stages share the same shape: multi-turn tool use, real (not simulated) environments, sparse outcome rewards, and GRPO, warm-started from the coding-boosted checkpoint. They run in order: <strong>SWE → Terminal → Search</strong>.</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/FxSMYasHnR4vpv263_DAL.png\"><picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/FxSMYasHnR4vpv263_DAL.png\" alt=\"Agentic RL environments\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture></a></p>\n<p><em>Figure 2. The three agentic-RL environments. Each pairs a real harness with a real environment and a sparse, outcome-based reward. The 3B model runs none of these.</em></p>\n<ul>\n<li><strong>SWE agent (software engineering).</strong> Each task is a real repository in its own sandbox. Driven by the <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://github.com/All-Hands-AI/OpenHands\">OpenHands</a> harness, the model reads code, edits files, and runs the test suite over many internal turns. The reward is verifiable: do the hidden tests pass? Tasks are drawn from open-source SWE datasets, each instance backed by a per-repo container image.</li>\n<li><strong>Terminal agent (terminal / OS operation).</strong> Multi-step tasks in a live shell, run through the Harbor / <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.harborframework.com/docs/agents/terminus-2\">Terminus-2</a> agent harness. The model plans a sequence of commands, observes their output, and recovers from errors. Reward is assigned when the task completed successfully. This is the one stage that drives its multi-turn agent loop at the GRPO level, with rollouts spanning up to 64 environment turns.</li>\n<li><strong>Search agent (deep research).</strong> The model answers hard, multi-hop questions using live web-search tool calls inside a browsing agent loop: gather evidence across hops, reason over it, and produce an answer. Because correctness here is open-ended, the reward is an LLM judge on the final answer.</li>\n</ul>\n<h3>Alignment: RLHF</h3>\n<p>The final stage of every model is <strong>RLHF for human preference and safety.</strong> It optimizes against a generative reward model (GenRM) for preference, plus a safety reward covering jailbreak resistance and appropriate refusals. This stage uses the highest KL penalty in the pipeline, aligning tone and safety without eroding the capabilities the earlier stages built. In addition to human preference and safety alignment, this stage also applies a reasoning-length penalty to discourage overly verbose reasoning behavior acquired during earlier stages.</p>\n<h3>How the Three Sizes Differ</h3>\n<p>Same method and infrastructure; the difference is how far up the ladder each model goes.</p>\n<table>\n<thead>\n<tr>\n<th>Stage</th>\n<th>3B</th>\n<th>8B</th>\n<th>30B</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td><strong>RLVR</strong> (verifiable)</td>\n<td>×2</td>\n<td>×2</td>\n<td>×3</td>\n</tr>\n<tr>\n<td><strong>Skill boosters</strong></td>\n<td>code</td>\n<td>IF · GPQA · code</td>\n<td>IF · code</td>\n</tr>\n<tr>\n<td><strong>SWE agent</strong></td>\n<td>—</td>\n<td>✓</td>\n<td>✓</td>\n</tr>\n<tr>\n<td><strong>Terminal agent</strong></td>\n<td>—</td>\n<td>✓</td>\n<td>✓</td>\n</tr>\n<tr>\n<td><strong>Search agent</strong></td>\n<td>—</td>\n<td>✓</td>\n<td>✓</td>\n</tr>\n<tr>\n<td><strong>RLHF</strong> (preference + safety)</td>\n<td>✓</td>\n<td>✓</td>\n<td>✓</td>\n</tr>\n</tbody>\n</table>\n<p>3B is a strong foundational-RL model; 8B and 30B add the agentic-RL block on top, learning to act with tools in real environments.</p>\n<hr />\n<h2>Agentic AI Infrastructure for Scalable RL</h2>\n<p>Reinforcement learning at this scale needs infrastructure that can drive a training loop and a fleet of live environments at the same time. This matters most in the agentic stages, where every training example is a multi-turn rollout that edits code, runs commands, or browses the web. Granite 4.2's RL runs on two open components: <strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://github.com/NVIDIA-NeMo/RL\">NeMo-RL</a></strong> on the training side and <strong><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://github.com/NVIDIA-NeMo/gym\">NeMo-Gym</a></strong> on the rollout side.</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/ceRxgjK0m0kVHp1iJ7P34.png\"><picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/ceRxgjK0m0kVHp1iJ7P34.png\" alt=\"NeMo-RL + NeMo-Gym system architecture\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture></a></p>\n<p><em>Figure 3. The RL system. NeMo-RL drives the GRPO loop (Megatron-Core training backend, vLLM generation, and Megatron-Bridge for HF⇄Megatron weight conversion). NeMo-Gym orchestrates rollouts and hosts the tools, sandboxes, and reward/verifier calls as pluggable <strong>Resources</strong>.</em></p>\n<p>The division of labor:</p>\n<ul>\n<li><strong>NeMo-RL (training side).</strong> <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://github.com/NVIDIA/Megatron-LM\">Megatron-Core</a> is the training backend; <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://github.com/vllm-project/vllm\">vLLM</a> generates rollouts; <strong>Megatron-Bridge</strong> converts weights between Megatron and Hugging Face formats, so each stage can export a clean HF checkpoint for the next one.</li>\n<li><strong>NeMo-Gym (rollout side).</strong> It exposes each environment as a set of <strong>Resources</strong> (verifiers, tools, sandboxes, and reward models) behind a uniform interface. This is the plug point for the agentic stages: the SWE repo sandboxes, the terminal harness, and the web-search tools all attach here, and to the training loop they look the same as a simple math verifier.</li>\n</ul>\n<p>That uniformity is what makes the staged curriculum above practical: a booster's rule-based checker and a full SWE sandbox present the same interface to GRPO.</p>\n<p>This split is also what makes the <strong>asynchronous</strong> training loop described above physically possible: generation and policy updates live on separate GPU pools, so the expensive generation fleet — including the live agentic environments — stays busy instead of idling through optimizer steps.</p>\n<hr />\n<h2>Results</h2>\n<p>Granite 4.2 was evaluated across agentic coding, general agentic and tool use, reasoning, chat and instruction following, and long context. The full benchmark table is below, followed by charts that break out the headline results by model size.</p>\n<table>\n<thead>\n<tr>\n<th>Task</th>\n<th>3B Dense</th>\n<th>8B Dense</th>\n<th>30B Dense</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Agentic (Coding)</td>\n<td></td>\n<td></td>\n<td></td>\n</tr>\n<tr>\n<td>SWE Bench Multilingual</td>\n<td>NA</td>\n<td>30.78</td>\n<td>41.89</td>\n</tr>\n<tr>\n<td>SWE Bench Pro</td>\n<td>NA</td>\n<td>19.11</td>\n<td>33.29</td>\n</tr>\n<tr>\n<td>SWE Bench Verified</td>\n<td>NA</td>\n<td>47.67</td>\n<td>57.00</td>\n</tr>\n<tr>\n<td>Terminal-Bench 2.1</td>\n<td>NA</td>\n<td>20.56</td>\n<td>29.24</td>\n</tr>\n<tr>\n<td>Agentic (General)</td>\n<td></td>\n<td></td>\n<td></td>\n</tr>\n<tr>\n<td>τ³-bench</td>\n<td>45.78</td>\n<td>58.06</td>\n<td>62.00</td>\n</tr>\n<tr>\n<td>BFCL (v4)</td>\n<td>52.41</td>\n<td>50.29</td>\n<td>61.39</td>\n</tr>\n<tr>\n<td>ProfBench</td>\n<td>32.10</td>\n<td>41.20</td>\n<td>42.90</td>\n</tr>\n<tr>\n<td>BirdBench</td>\n<td>NA</td>\n<td>41.07</td>\n<td>41.85</td>\n</tr>\n<tr>\n<td>GDPval</td>\n<td>NA</td>\n<td>1189.00</td>\n<td>1225.00</td>\n</tr>\n<tr>\n<td>Reasoning</td>\n<td></td>\n<td></td>\n<td></td>\n</tr>\n<tr>\n<td>AIME25</td>\n<td>78.33</td>\n<td>86.67</td>\n<td>89.17</td>\n</tr>\n<tr>\n<td>HMMT Feb25</td>\n<td>66.67</td>\n<td>78.33</td>\n<td>89.17</td>\n</tr>\n<tr>\n<td>GPQA</td>\n<td>54.80</td>\n<td>64.14</td>\n<td>66.41</td>\n</tr>\n<tr>\n<td>LiveCodeBench v6</td>\n<td>69.71</td>\n<td>73.24</td>\n<td>75.77</td>\n</tr>\n<tr>\n<td>SciCode</td>\n<td>24.11</td>\n<td>36.09</td>\n<td>38.76</td>\n</tr>\n<tr>\n<td>Chat &amp; Instruction Following</td>\n<td></td>\n<td></td>\n<td></td>\n</tr>\n<tr>\n<td>MMLU-Pro</td>\n<td>67.84</td>\n<td>74.04</td>\n<td>77.60</td>\n</tr>\n<tr>\n<td>MMLU-ProX lite (IBM)</td>\n<td>27.78</td>\n<td>61.06</td>\n<td>66.64</td>\n</tr>\n<tr>\n<td>Arena-Hard-V2</td>\n<td>34.96</td>\n<td>65.19</td>\n<td>67.93</td>\n</tr>\n<tr>\n<td>IFBench (prompt)</td>\n<td>74.33</td>\n<td>79.33</td>\n<td>77.17</td>\n</tr>\n<tr>\n<td>Long Context</td>\n<td></td>\n<td></td>\n<td></td>\n</tr>\n<tr>\n<td>RULER 64K</td>\n<td>67.52</td>\n<td>80.99</td>\n<td>89.96</td>\n</tr>\n<tr>\n<td>RULER 128K</td>\n<td>55.30</td>\n<td>71.41</td>\n<td>81.38</td>\n</tr>\n</tbody>\n</table>\n<p><strong>Supported languages:</strong> English, German, Spanish, French, Japanese, Portuguese, Arabic, Czech, Italian, Korean, Dutch, and Chinese.</p>\n<p>The charts below break these results out by capability area.</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/u4p7x5-U_7Qj9w-ovsZ7D.png\"><picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/u4p7x5-U_7Qj9w-ovsZ7D.png\" alt=\"Reasoning benchmarks\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture></a></p>\n<p><em>Figure 4. Reasoning (pass@1). Scores rise consistently with model size across math (AIME25, HMMT), science (GPQA), and code reasoning (LiveCodeBench, SciCode).</em></p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/CH5Aqnjba9hHlBteGNvw0.png\"><picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/CH5Aqnjba9hHlBteGNvw0.png\" alt=\"Agentic coding benchmarks\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture></a></p>\n<p><em>Figure 5. Agentic coding resolve rates. The agentic-RL block is trained only for 8B and 30B; the 30B model leads across SWE-Bench variants and Terminal-Bench.</em></p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/mXZuw86XqEvna7Rm2OcTi.png\"><picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/mXZuw86XqEvna7Rm2OcTi.png\" alt=\"General agentic and tool-use benchmarks\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture></a></p>\n<p><em>Figure 6. General agentic and tool-use benchmarks, reported for all three sizes.</em></p>\n<hr />\n<h2>Quantization</h2>\n<p>We also released four quantized variants of the Granite 4.2 models for inference with vLLM. The models are converted to FP8, NVFP4, and MXFP4 using LLM Compressor, and to the GGUF format using the llama.cpp framework for reduced-memory deployment.</p>\n<h3>FP8</h3>\n<p>The FP8 version is quantized with dynamic per-channel weights and per-token activations. No calibration is used.</p>\n<h3>FP4</h3>\n<p>The NVFP4 and MXFP4 versions are quantized using GPTQ calibrated on 2K samples drawn from the SFT dataset. Max context length is 2K during calibration.</p>\n<h3>GGUF</h3>\n<p>Conversion of the Granite 4.2 models to GGUF is done with the canonical llama.cpp tool as described in <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://github.com/IBM/gguf#gguf-conversion--quantization\">https://github.com/IBM/gguf#gguf-conversion--quantization</a>.</p>\n<p>Several GGUF formats are provided:</p>\n<ul>\n<li>Q8_0</li>\n<li>Q6_K</li>\n<li>Q5_K_S</li>\n<li>Q5_K_M</li>\n<li>Q5_1</li>\n<li>Q5_0</li>\n<li>Q4_K_S</li>\n<li>Q4_K_M</li>\n<li>Q4_1</li>\n<li>Q4_0</li>\n<li>Q3_K_S</li>\n<li>Q3_K_M</li>\n<li>Q3_K_L</li>\n<li>Q2_K</li>\n</ul>\n<hr />\n<h2>Infrastructure</h2>\n<h3>Hardware</h3>\n<p>We trained the Granite 4.2 language models on an NVIDIA GB200 NVL72 cluster hosted by CoreWeave, featuring:</p>\n<ul>\n<li>A 72-GPU NVLink domain for high-speed intra-rack communication</li>\n<li>A non-blocking Fat-Tree NDR 400 Gb/s InfiniBand fabric for full-bandwidth inter-rack connectivity</li>\n<li>Thousands of GPUs operating at cluster scale</li>\n</ul>\n<p>This infrastructure delivers the high-bandwidth, low-latency communication required for efficient large-scale distributed training.</p>\n<h3>Software Stack</h3>\n<p>The training software stack is packaged into <code>.sqsh</code> container images, each giving a run a reproducible, portable environment with its SBSA-compatible CUDA targets, Linux aarch64 Python wheels, and GPU-specific binaries pinned. The large-scale SFT runs build on an NGC PyTorch base image (Ubuntu 22.04, CUDA 12.8, Python 3.12); the RL stack runs in its own NeMo-RL container.</p>\n<hr />\n<h2>Getting Started (Transformers)</h2>\n<h3>Installation</h3>\n<pre><code class=\"language-bash\"><span class=\"code-snippet\" data-language=\"Shell\"><span class=\"syntax-source syntax-shell syntax-bash\"><span class=\"syntax-meta syntax-function-call syntax-shell\"><span class=\"syntax-variable syntax-function syntax-shell\">pip</span></span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-shell\"> install torch</span>\n<span class=\"syntax-meta syntax-function-call syntax-shell\"><span class=\"syntax-variable syntax-function syntax-shell\">pip</span></span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-shell\"> install accelerate transformers</span>\n</span></span></code></pre>\n<h3>Basic Inference (Thinking Mode)</h3>\n<pre><code class=\"language-python\"><span class=\"code-snippet\" data-language=\"Python\"><span class=\"syntax-source syntax-python\"><span class=\"syntax-meta syntax-statement syntax-import syntax-python\"><span class=\"syntax-keyword syntax-control syntax-import syntax-python\">import</span> <span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">torch</span></span></span>\n<span class=\"syntax-meta syntax-statement syntax-import syntax-python\"><span class=\"syntax-keyword syntax-control syntax-import syntax-from syntax-python\">from</span></span><span class=\"syntax-meta syntax-statement syntax-import syntax-python\"><span class=\"syntax-meta syntax-import-source syntax-python\"> <span class=\"syntax-meta syntax-import-path syntax-python\"><span class=\"syntax-meta syntax-import-name syntax-python\">transformers</span></span> <span class=\"syntax-meta syntax-statement syntax-import syntax-python\"><span class=\"syntax-keyword syntax-control syntax-import syntax-python\">import</span></span></span></span><span class=\"syntax-meta syntax-statement syntax-import syntax-python\"></span><span class=\"syntax-meta syntax-statement syntax-import syntax-python\"> <span class=\"syntax-meta syntax-generic-name syntax-python\">AutoModelForCausalLM</span><span class=\"syntax-punctuation syntax-separator syntax-import-list syntax-python\">,</span> <span class=\"syntax-meta syntax-generic-name syntax-python\">AutoTokenizer</span></span>\n\n<span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">model_path</span></span> <span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">ibm-granite/granite-4.2-30b<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span>\n<span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">tokenizer</span></span> <span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span> <span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">AutoTokenizer</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span></span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">from_pretrained</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">model_path</span></span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span>\n<span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">model</span></span> <span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span> <span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">AutoModelForCausalLM</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span></span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">from_pretrained</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">model_path</span></span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">device_map</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">cuda<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">torch_dtype</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">torch</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span><span class=\"syntax-meta syntax-generic-name syntax-python\">bfloat16</span></span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span>\n<span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">model</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span></span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">eval</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span>\n\n<span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">messages</span></span> <span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span> <span class=\"syntax-meta syntax-structure syntax-list syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-list syntax-begin syntax-python\">[</span>\n    <span class=\"syntax-meta syntax-structure syntax-dictionary-or-set syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-dictionary-or-set syntax-begin syntax-python\">{</span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">role<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-python\">:</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">user<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-dictionary-or-set syntax-python\">,</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">content<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-python\">:</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">How many r&#39;s are in the word &#39;strawberry&#39;?<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-section syntax-dictionary-or-set syntax-end syntax-python\">}</span></span><span class=\"syntax-punctuation syntax-separator syntax-list syntax-python\">,</span>\n<span class=\"syntax-punctuation syntax-section syntax-list syntax-end syntax-python\">]</span></span>\n\n<span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">text</span></span> <span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span> <span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">tokenizer</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span></span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">apply_chat_template</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">messages</span></span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">tokenize</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-language syntax-python\">False</span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">add_generation_prompt</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-language syntax-python\">True</span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">enable_thinking</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-language syntax-python\">True</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span>\n<span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">inputs</span></span> <span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span> <span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">tokenizer</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">text</span></span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">return_tensors</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">pt<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span><span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span></span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">to</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">model</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span><span class=\"syntax-meta syntax-generic-name syntax-python\">device</span></span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span>\n\n<span class=\"syntax-meta syntax-statement syntax-with syntax-python\"><span class=\"syntax-keyword syntax-control syntax-flow syntax-with syntax-python\">with</span> <span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">torch</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span></span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">no_grad</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span><span class=\"syntax-punctuation syntax-section syntax-block syntax-with syntax-python\">:</span></span>\n    <span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">output</span></span> <span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span> <span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">model</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span></span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">generate</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-keyword syntax-operator syntax-arithmetic syntax-python\">*</span><span class=\"syntax-keyword syntax-operator syntax-arithmetic syntax-python\">*</span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">inputs</span></span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">max_new_tokens</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-numeric syntax-integer syntax-decimal syntax-python\">8192</span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">temperature</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-numeric syntax-float syntax-python\">1<span class=\"syntax-punctuation syntax-separator syntax-decimal syntax-python\">.</span>0</span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">top_p</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-numeric syntax-float syntax-python\">0<span class=\"syntax-punctuation syntax-separator syntax-decimal syntax-python\">.</span>95</span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">do_sample</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-language syntax-python\">True</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span>\n\n<span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-support syntax-function syntax-builtin syntax-python\">print</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">tokenizer</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span></span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">decode</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">output</span></span></span><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-brackets syntax-begin syntax-python\">[</span></span><span class=\"syntax-meta syntax-item-access syntax-arguments syntax-python\"><span class=\"syntax-constant syntax-numeric syntax-integer syntax-decimal syntax-python\">0</span></span><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-brackets syntax-end syntax-python\">]</span></span><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-brackets syntax-begin syntax-python\">[</span></span><span class=\"syntax-meta syntax-item-access syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">inputs</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span><span class=\"syntax-meta syntax-generic-name syntax-python\">input_ids</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span><span class=\"syntax-meta syntax-generic-name syntax-python\">shape</span></span></span><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-brackets syntax-begin syntax-python\">[</span></span><span class=\"syntax-meta syntax-item-access syntax-arguments syntax-python\"><span class=\"syntax-keyword syntax-operator syntax-arithmetic syntax-python\">-</span><span class=\"syntax-constant syntax-numeric syntax-integer syntax-decimal syntax-python\">1</span></span><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-brackets syntax-end syntax-python\">]</span></span><span class=\"syntax-punctuation syntax-separator syntax-slice syntax-python\">:</span></span><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-brackets syntax-end syntax-python\">]</span></span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">skip_special_tokens</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-language syntax-python\">False</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span>\n</span></span></code></pre>\n<p><strong>Example Output</strong></p>\n<pre><code><span class=\"code-snippet\">&lt;think&gt;\nOkay, let&#39;s see. The problem is to find how many &#39;r&#39;s are in the word &#39;strawberry&#39;.\n\nFirst, I need to write out the word: s t r a w b e r r y.\n\nNow, I need to count the number of &#39;r&#39; letters. Let&#39;s list each letter and check for &#39;r&#39;.\n\n1. s – not r\n2. t – not r\n3. r – yes, that&#39;s one\n4. a – no\n5. w – no\n6. b – no\n7. e – no\n8. r – yes, that&#39;s two\n9. r – yes, that&#39;s three\n10. y – no\n\nTotal r&#39;s = 3.\n&lt;/think&gt;\nThere are **3** r&#39;s in the word &quot;strawberry&quot;.&lt;|im_end|&gt;\n</span></code></pre>\n<h3>Non-Thinking Mode</h3>\n<pre><code class=\"language-python\"><span class=\"code-snippet\" data-language=\"Python\"><span class=\"syntax-source syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">messages</span></span> <span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span> <span class=\"syntax-meta syntax-structure syntax-list syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-list syntax-begin syntax-python\">[</span>\n    <span class=\"syntax-meta syntax-structure syntax-dictionary-or-set syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-dictionary-or-set syntax-begin syntax-python\">{</span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">role<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-python\">:</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">user<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-dictionary-or-set syntax-python\">,</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">content<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-python\">:</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">What is the capital of France?<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-section syntax-dictionary-or-set syntax-end syntax-python\">}</span></span><span class=\"syntax-punctuation syntax-separator syntax-list syntax-python\">,</span>\n<span class=\"syntax-punctuation syntax-section syntax-list syntax-end syntax-python\">]</span></span>\n<span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">text</span></span> <span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span> <span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">tokenizer</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span></span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">apply_chat_template</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">messages</span></span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">tokenize</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-language syntax-python\">False</span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">add_generation_prompt</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-language syntax-python\">True</span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">enable_thinking</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-language syntax-python\">False</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span>\n<span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">inputs</span></span> <span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span> <span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">tokenizer</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">text</span></span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">return_tensors</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">pt<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span><span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span></span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">to</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">model</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span><span class=\"syntax-meta syntax-generic-name syntax-python\">device</span></span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span>\n\n<span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">output</span></span> <span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span> <span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">model</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span></span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">generate</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-keyword syntax-operator syntax-arithmetic syntax-python\">*</span><span class=\"syntax-keyword syntax-operator syntax-arithmetic syntax-python\">*</span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">inputs</span></span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">max_new_tokens</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-numeric syntax-integer syntax-decimal syntax-python\">2048</span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">temperature</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-numeric syntax-float syntax-python\">1<span class=\"syntax-punctuation syntax-separator syntax-decimal syntax-python\">.</span>0</span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">top_p</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-numeric syntax-float syntax-python\">0<span class=\"syntax-punctuation syntax-separator syntax-decimal syntax-python\">.</span>95</span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">do_sample</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-language syntax-python\">True</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span>\n<span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-support syntax-function syntax-builtin syntax-python\">print</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">tokenizer</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span></span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">decode</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">output</span></span></span><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-brackets syntax-begin syntax-python\">[</span></span><span class=\"syntax-meta syntax-item-access syntax-arguments syntax-python\"><span class=\"syntax-constant syntax-numeric syntax-integer syntax-decimal syntax-python\">0</span></span><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-brackets syntax-end syntax-python\">]</span></span><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-brackets syntax-begin syntax-python\">[</span></span><span class=\"syntax-meta syntax-item-access syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">inputs</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span><span class=\"syntax-meta syntax-generic-name syntax-python\">input_ids</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span><span class=\"syntax-meta syntax-generic-name syntax-python\">shape</span></span></span><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-brackets syntax-begin syntax-python\">[</span></span><span class=\"syntax-meta syntax-item-access syntax-arguments syntax-python\"><span class=\"syntax-keyword syntax-operator syntax-arithmetic syntax-python\">-</span><span class=\"syntax-constant syntax-numeric syntax-integer syntax-decimal syntax-python\">1</span></span><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-brackets syntax-end syntax-python\">]</span></span><span class=\"syntax-punctuation syntax-separator syntax-slice syntax-python\">:</span></span><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-brackets syntax-end syntax-python\">]</span></span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">skip_special_tokens</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-language syntax-python\">False</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span>\n</span></span></code></pre>\n<p><strong>Example Output</strong></p>\n<pre><code><span class=\"code-snippet\">&lt;think&gt;&lt;/think&gt;The capital of France is Paris.&lt;|im_end|&gt;\n</span></code></pre>\n<h3>Low-Effort Thinking</h3>\n<pre><code class=\"language-python\"><span class=\"code-snippet\" data-language=\"Python\"><span class=\"syntax-source syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">messages</span></span> <span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span> <span class=\"syntax-meta syntax-structure syntax-list syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-list syntax-begin syntax-python\">[</span>\n    <span class=\"syntax-meta syntax-structure syntax-dictionary-or-set syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-dictionary-or-set syntax-begin syntax-python\">{</span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">role<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-python\">:</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">user<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-dictionary-or-set syntax-python\">,</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">content<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-python\">:</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">What is 2 + 2?<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-section syntax-dictionary-or-set syntax-end syntax-python\">}</span></span><span class=\"syntax-punctuation syntax-separator syntax-list syntax-python\">,</span>\n<span class=\"syntax-punctuation syntax-section syntax-list syntax-end syntax-python\">]</span></span>\n<span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">text</span></span> <span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span> <span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">tokenizer</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span></span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">apply_chat_template</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">messages</span></span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">tokenize</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-language syntax-python\">False</span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">add_generation_prompt</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-language syntax-python\">True</span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span>\n                                     <span class=\"syntax-variable syntax-parameter syntax-python\">enable_thinking</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-language syntax-python\">True</span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">low_effort</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-language syntax-python\">True</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span>\n<span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">inputs</span></span> <span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span> <span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">tokenizer</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">text</span></span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">return_tensors</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">pt<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span><span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span></span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">to</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">model</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span><span class=\"syntax-meta syntax-generic-name syntax-python\">device</span></span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span>\n\n<span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">output</span></span> <span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span> <span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">model</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span></span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">generate</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-keyword syntax-operator syntax-arithmetic syntax-python\">*</span><span class=\"syntax-keyword syntax-operator syntax-arithmetic syntax-python\">*</span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">inputs</span></span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">max_new_tokens</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-numeric syntax-integer syntax-decimal syntax-python\">4096</span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">temperature</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-numeric syntax-float syntax-python\">1<span class=\"syntax-punctuation syntax-separator syntax-decimal syntax-python\">.</span>0</span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">top_p</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-numeric syntax-float syntax-python\">0<span class=\"syntax-punctuation syntax-separator syntax-decimal syntax-python\">.</span>95</span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">do_sample</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-language syntax-python\">True</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span>\n<span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-support syntax-function syntax-builtin syntax-python\">print</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">tokenizer</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span></span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">decode</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">output</span></span></span><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-brackets syntax-begin syntax-python\">[</span></span><span class=\"syntax-meta syntax-item-access syntax-arguments syntax-python\"><span class=\"syntax-constant syntax-numeric syntax-integer syntax-decimal syntax-python\">0</span></span><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-brackets syntax-end syntax-python\">]</span></span><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-brackets syntax-begin syntax-python\">[</span></span><span class=\"syntax-meta syntax-item-access syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">inputs</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span><span class=\"syntax-meta syntax-generic-name syntax-python\">input_ids</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span><span class=\"syntax-meta syntax-generic-name syntax-python\">shape</span></span></span><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-brackets syntax-begin syntax-python\">[</span></span><span class=\"syntax-meta syntax-item-access syntax-arguments syntax-python\"><span class=\"syntax-keyword syntax-operator syntax-arithmetic syntax-python\">-</span><span class=\"syntax-constant syntax-numeric syntax-integer syntax-decimal syntax-python\">1</span></span><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-brackets syntax-end syntax-python\">]</span></span><span class=\"syntax-punctuation syntax-separator syntax-slice syntax-python\">:</span></span><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-brackets syntax-end syntax-python\">]</span></span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">skip_special_tokens</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-language syntax-python\">False</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span>\n</span></span></code></pre>\n<p><strong>Example Output</strong></p>\n<pre><code><span class=\"code-snippet\">&lt;think&gt;\nSimple answer.\n&lt;/think&gt;\n2 + 2 = 4.&lt;|im_end|&gt;\n</span></code></pre>\n<hr />\n<h2>Tool Calling</h2>\n<p>Granite models support tool calling with integrated reasoning: the model reasons about which tool to call and why before calling it. Tools are defined with the <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://platform.openai.com/docs/guides/function-calling\">OpenAI function definition schema</a>.</p>\n<h3>Basic Tool Calling</h3>\n<pre><code class=\"language-python\"><span class=\"code-snippet\" data-language=\"Python\"><span class=\"syntax-source syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">tools</span></span> <span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span> <span class=\"syntax-meta syntax-structure syntax-list syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-list syntax-begin syntax-python\">[</span>\n    <span class=\"syntax-meta syntax-structure syntax-dictionary-or-set syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-dictionary-or-set syntax-begin syntax-python\">{</span>\n        <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">type<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-python\">:</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">function<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-dictionary-or-set syntax-python\">,</span>\n        <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">function<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-python\">:</span> <span class=\"syntax-meta syntax-structure syntax-dictionary-or-set syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-dictionary-or-set syntax-begin syntax-python\">{</span>\n            <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">name<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-python\">:</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">get_current_weather<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-dictionary-or-set syntax-python\">,</span>\n            <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">description<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-python\">:</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">Get the current weather for a specified city.<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-dictionary-or-set syntax-python\">,</span>\n            <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">parameters<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-python\">:</span> <span class=\"syntax-meta syntax-structure syntax-dictionary-or-set syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-dictionary-or-set syntax-begin syntax-python\">{</span>\n                <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">type<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-python\">:</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">object<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-dictionary-or-set syntax-python\">,</span>\n                <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">properties<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-python\">:</span> <span class=\"syntax-meta syntax-structure syntax-dictionary-or-set syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-dictionary-or-set syntax-begin syntax-python\">{</span>\n                    <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">city<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-python\">:</span> <span class=\"syntax-meta syntax-structure syntax-dictionary-or-set syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-dictionary-or-set syntax-begin syntax-python\">{</span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">type<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-python\">:</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">string<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-dictionary-or-set syntax-python\">,</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">description<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-python\">:</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">Name of the city<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-section syntax-dictionary-or-set syntax-end syntax-python\">}</span></span>\n                <span class=\"syntax-punctuation syntax-section syntax-dictionary-or-set syntax-end syntax-python\">}</span></span><span class=\"syntax-punctuation syntax-separator syntax-dictionary-or-set syntax-python\">,</span>\n                <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">required<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-python\">:</span> <span class=\"syntax-meta syntax-structure syntax-list syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-list syntax-begin syntax-python\">[</span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">city<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-section syntax-list syntax-end syntax-python\">]</span></span>\n            <span class=\"syntax-punctuation syntax-section syntax-dictionary-or-set syntax-end syntax-python\">}</span></span>\n        <span class=\"syntax-punctuation syntax-section syntax-dictionary-or-set syntax-end syntax-python\">}</span></span>\n    <span class=\"syntax-punctuation syntax-section syntax-dictionary-or-set syntax-end syntax-python\">}</span></span>\n<span class=\"syntax-punctuation syntax-section syntax-list syntax-end syntax-python\">]</span></span>\n\n<span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">messages</span></span> <span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span> <span class=\"syntax-meta syntax-structure syntax-list syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-list syntax-begin syntax-python\">[</span>\n    <span class=\"syntax-meta syntax-structure syntax-dictionary-or-set syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-dictionary-or-set syntax-begin syntax-python\">{</span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">role<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-python\">:</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">user<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-dictionary-or-set syntax-python\">,</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">content<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-python\">:</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">What&#39;s the weather like in Boston right now?<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-section syntax-dictionary-or-set syntax-end syntax-python\">}</span></span><span class=\"syntax-punctuation syntax-separator syntax-list syntax-python\">,</span>\n<span class=\"syntax-punctuation syntax-section syntax-list syntax-end syntax-python\">]</span></span>\n<span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">text</span></span> <span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span> <span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">tokenizer</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span></span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">apply_chat_template</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">messages</span></span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">tokenize</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-language syntax-python\">False</span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">tools</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">tools</span></span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span>\n                                     <span class=\"syntax-variable syntax-parameter syntax-python\">add_generation_prompt</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-language syntax-python\">True</span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">enable_thinking</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-language syntax-python\">True</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span>\n<span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">inputs</span></span> <span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span> <span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">tokenizer</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">text</span></span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">return_tensors</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">pt<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span><span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span></span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">to</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">model</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span><span class=\"syntax-meta syntax-generic-name syntax-python\">device</span></span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span>\n\n<span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">output</span></span> <span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span> <span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">model</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span></span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">generate</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-keyword syntax-operator syntax-arithmetic syntax-python\">*</span><span class=\"syntax-keyword syntax-operator syntax-arithmetic syntax-python\">*</span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">inputs</span></span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">max_new_tokens</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-numeric syntax-integer syntax-decimal syntax-python\">4096</span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">temperature</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-numeric syntax-float syntax-python\">1<span class=\"syntax-punctuation syntax-separator syntax-decimal syntax-python\">.</span>0</span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">top_p</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-numeric syntax-float syntax-python\">0<span class=\"syntax-punctuation syntax-separator syntax-decimal syntax-python\">.</span>95</span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">do_sample</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-language syntax-python\">True</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span>\n<span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-support syntax-function syntax-builtin syntax-python\">print</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">tokenizer</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span></span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">decode</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">output</span></span></span><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-brackets syntax-begin syntax-python\">[</span></span><span class=\"syntax-meta syntax-item-access syntax-arguments syntax-python\"><span class=\"syntax-constant syntax-numeric syntax-integer syntax-decimal syntax-python\">0</span></span><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-brackets syntax-end syntax-python\">]</span></span><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-brackets syntax-begin syntax-python\">[</span></span><span class=\"syntax-meta syntax-item-access syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">inputs</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span><span class=\"syntax-meta syntax-generic-name syntax-python\">input_ids</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span><span class=\"syntax-meta syntax-generic-name syntax-python\">shape</span></span></span><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-brackets syntax-begin syntax-python\">[</span></span><span class=\"syntax-meta syntax-item-access syntax-arguments syntax-python\"><span class=\"syntax-keyword syntax-operator syntax-arithmetic syntax-python\">-</span><span class=\"syntax-constant syntax-numeric syntax-integer syntax-decimal syntax-python\">1</span></span><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-brackets syntax-end syntax-python\">]</span></span><span class=\"syntax-punctuation syntax-separator syntax-slice syntax-python\">:</span></span><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-brackets syntax-end syntax-python\">]</span></span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">skip_special_tokens</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-language syntax-python\">False</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span>\n</span></span></code></pre>\n<p><strong>Example Output</strong></p>\n<pre><code><span class=\"code-snippet\">&lt;think&gt;\nOkay, the user is asking for the weather in Boston right now. Let me check the tools\navailable. There&#39;s a function called get_current_weather that takes a city parameter.\nI need to call that with the city set to Boston.\n&lt;/think&gt;\n&lt;tool_call&gt;\n&lt;function=get_current_weather&gt;\n&lt;parameter=city&gt;\nBoston\n&lt;/parameter&gt;\n&lt;/function&gt;\n&lt;/tool_call&gt;\n&lt;|im_end|&gt;\n</span></code></pre>\n<h3>Multi-Turn with Tool Response</h3>\n<pre><code class=\"language-python\"><span class=\"code-snippet\" data-language=\"Python\"><span class=\"syntax-source syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">messages</span></span> <span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span> <span class=\"syntax-meta syntax-structure syntax-list syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-list syntax-begin syntax-python\">[</span>\n    <span class=\"syntax-meta syntax-structure syntax-dictionary-or-set syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-dictionary-or-set syntax-begin syntax-python\">{</span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">role<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-python\">:</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">user<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-dictionary-or-set syntax-python\">,</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">content<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-python\">:</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">What&#39;s the weather like in Boston right now?<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-section syntax-dictionary-or-set syntax-end syntax-python\">}</span></span><span class=\"syntax-punctuation syntax-separator syntax-list syntax-python\">,</span>\n    <span class=\"syntax-meta syntax-structure syntax-dictionary-or-set syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-dictionary-or-set syntax-begin syntax-python\">{</span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">role<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-python\">:</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">assistant<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-dictionary-or-set syntax-python\">,</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">content<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-python\">:</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">&lt;think&gt;<span class=\"syntax-constant syntax-character syntax-escape syntax-python\">\\n</span>The user wants to know the current weather in Boston. I should call get_current_weather.<span class=\"syntax-constant syntax-character syntax-escape syntax-python\">\\n</span>&lt;/think&gt;<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-dictionary-or-set syntax-python\">,</span>\n     <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">tool_calls<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-python\">:</span> <span class=\"syntax-meta syntax-structure syntax-list syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-list syntax-begin syntax-python\">[</span><span class=\"syntax-meta syntax-structure syntax-dictionary-or-set syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-dictionary-or-set syntax-begin syntax-python\">{</span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">function<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-python\">:</span> <span class=\"syntax-meta syntax-structure syntax-dictionary-or-set syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-dictionary-or-set syntax-begin syntax-python\">{</span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">name<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-python\">:</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">get_current_weather<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-dictionary-or-set syntax-python\">,</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">arguments<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-python\">:</span> <span class=\"syntax-meta syntax-structure syntax-dictionary-or-set syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-dictionary-or-set syntax-begin syntax-python\">{</span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">city<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-python\">:</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">Boston<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-section syntax-dictionary-or-set syntax-end syntax-python\">}</span></span><span class=\"syntax-punctuation syntax-section syntax-dictionary-or-set syntax-end syntax-python\">}</span></span><span class=\"syntax-punctuation syntax-section syntax-dictionary-or-set syntax-end syntax-python\">}</span></span><span class=\"syntax-punctuation syntax-section syntax-list syntax-end syntax-python\">]</span></span><span class=\"syntax-punctuation syntax-section syntax-dictionary-or-set syntax-end syntax-python\">}</span></span><span class=\"syntax-punctuation syntax-separator syntax-list syntax-python\">,</span>\n    <span class=\"syntax-meta syntax-structure syntax-dictionary-or-set syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-dictionary-or-set syntax-begin syntax-python\">{</span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">role<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-python\">:</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">tool<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-dictionary-or-set syntax-python\">,</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">content<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-python\">:</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-single syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&#39;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-single syntax-python\">{&quot;temperature&quot;: &quot;72°F&quot;, &quot;condition&quot;: &quot;Partly cloudy&quot;, &quot;humidity&quot;: &quot;65%&quot;}<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&#39;</span></span></span><span class=\"syntax-punctuation syntax-section syntax-dictionary-or-set syntax-end syntax-python\">}</span></span><span class=\"syntax-punctuation syntax-separator syntax-list syntax-python\">,</span>\n<span class=\"syntax-punctuation syntax-section syntax-list syntax-end syntax-python\">]</span></span>\n<span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">text</span></span> <span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span> <span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">tokenizer</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span></span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">apply_chat_template</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">messages</span></span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">tokenize</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-language syntax-python\">False</span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">tools</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">tools</span></span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span>\n                                     <span class=\"syntax-variable syntax-parameter syntax-python\">add_generation_prompt</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-language syntax-python\">True</span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">enable_thinking</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-language syntax-python\">True</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span>\n<span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">inputs</span></span> <span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span> <span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">tokenizer</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">text</span></span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">return_tensors</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">pt<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span><span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span></span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">to</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">model</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span><span class=\"syntax-meta syntax-generic-name syntax-python\">device</span></span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span>\n\n<span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">output</span></span> <span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span> <span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">model</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span></span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">generate</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-keyword syntax-operator syntax-arithmetic syntax-python\">*</span><span class=\"syntax-keyword syntax-operator syntax-arithmetic syntax-python\">*</span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">inputs</span></span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">max_new_tokens</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-numeric syntax-integer syntax-decimal syntax-python\">4096</span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">temperature</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-numeric syntax-float syntax-python\">1<span class=\"syntax-punctuation syntax-separator syntax-decimal syntax-python\">.</span>0</span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">top_p</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-numeric syntax-float syntax-python\">0<span class=\"syntax-punctuation syntax-separator syntax-decimal syntax-python\">.</span>95</span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">do_sample</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-language syntax-python\">True</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span>\n<span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-support syntax-function syntax-builtin syntax-python\">print</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">tokenizer</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span></span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">decode</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">output</span></span></span><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-brackets syntax-begin syntax-python\">[</span></span><span class=\"syntax-meta syntax-item-access syntax-arguments syntax-python\"><span class=\"syntax-constant syntax-numeric syntax-integer syntax-decimal syntax-python\">0</span></span><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-brackets syntax-end syntax-python\">]</span></span><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-brackets syntax-begin syntax-python\">[</span></span><span class=\"syntax-meta syntax-item-access syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">inputs</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span><span class=\"syntax-meta syntax-generic-name syntax-python\">input_ids</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span><span class=\"syntax-meta syntax-generic-name syntax-python\">shape</span></span></span><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-brackets syntax-begin syntax-python\">[</span></span><span class=\"syntax-meta syntax-item-access syntax-arguments syntax-python\"><span class=\"syntax-keyword syntax-operator syntax-arithmetic syntax-python\">-</span><span class=\"syntax-constant syntax-numeric syntax-integer syntax-decimal syntax-python\">1</span></span><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-brackets syntax-end syntax-python\">]</span></span><span class=\"syntax-punctuation syntax-separator syntax-slice syntax-python\">:</span></span><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-brackets syntax-end syntax-python\">]</span></span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">skip_special_tokens</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-language syntax-python\">False</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span>\n</span></span></code></pre>\n<p><strong>Example Output</strong></p>\n<pre><code><span class=\"code-snippet\">&lt;think&gt;\nThe tool returned the weather data for Boston: temperature 72°F, partly cloudy, humidity 65%.\nI need to present this information clearly to the user.\n&lt;/think&gt;\nThe current weather in Boston is 72°F, partly cloudy, with 65% humidity.&lt;|im_end|&gt;\n</span></code></pre>\n<hr />\n<h2>Multi-Turn Conversations</h2>\n<h3>History Thinking Truncation</h3>\n<pre><code class=\"language-python\"><span class=\"code-snippet\" data-language=\"Python\"><span class=\"syntax-source syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">messages</span></span> <span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span> <span class=\"syntax-meta syntax-structure syntax-list syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-list syntax-begin syntax-python\">[</span>\n    <span class=\"syntax-meta syntax-structure syntax-dictionary-or-set syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-dictionary-or-set syntax-begin syntax-python\">{</span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">role<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-python\">:</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">user<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-dictionary-or-set syntax-python\">,</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">content<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-python\">:</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">What is 15 * 37?<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-section syntax-dictionary-or-set syntax-end syntax-python\">}</span></span><span class=\"syntax-punctuation syntax-separator syntax-list syntax-python\">,</span>\n    <span class=\"syntax-meta syntax-structure syntax-dictionary-or-set syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-dictionary-or-set syntax-begin syntax-python\">{</span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">role<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-python\">:</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">assistant<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-dictionary-or-set syntax-python\">,</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">content<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-python\">:</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">&lt;think&gt;<span class=\"syntax-constant syntax-character syntax-escape syntax-python\">\\n</span>Let me calculate 15 * 37.<span class=\"syntax-constant syntax-character syntax-escape syntax-python\">\\n</span>15 * 37 = 15 * 30 + 15 * 7 = 450 + 105 = 555<span class=\"syntax-constant syntax-character syntax-escape syntax-python\">\\n</span>&lt;/think&gt;<span class=\"syntax-constant syntax-character syntax-escape syntax-python\">\\n</span>15 * 37 = 555<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-section syntax-dictionary-or-set syntax-end syntax-python\">}</span></span><span class=\"syntax-punctuation syntax-separator syntax-list syntax-python\">,</span>\n    <span class=\"syntax-meta syntax-structure syntax-dictionary-or-set syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-dictionary-or-set syntax-begin syntax-python\">{</span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">role<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-python\">:</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">user<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-dictionary-or-set syntax-python\">,</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">content<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-python\">:</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\">Now divide that by 5<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span><span class=\"syntax-punctuation syntax-section syntax-dictionary-or-set syntax-end syntax-python\">}</span></span><span class=\"syntax-punctuation syntax-separator syntax-list syntax-python\">,</span>\n<span class=\"syntax-punctuation syntax-section syntax-list syntax-end syntax-python\">]</span></span>\n\n<span class=\"syntax-comment syntax-line syntax-number-sign syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-comment syntax-python\">#</span> Default: previous thinking is stripped to save context\n</span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">text</span></span> <span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span> <span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">tokenizer</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span></span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">apply_chat_template</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">messages</span></span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">tokenize</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-language syntax-python\">False</span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">add_generation_prompt</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-language syntax-python\">True</span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span>\n                                     <span class=\"syntax-variable syntax-parameter syntax-python\">enable_thinking</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-language syntax-python\">True</span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">truncate_history_thinking</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-language syntax-python\">True</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span>\n\n<span class=\"syntax-comment syntax-line syntax-number-sign syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-comment syntax-python\">#</span> To preserve full history:\n</span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">text_full</span></span> <span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span> <span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">tokenizer</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span></span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">apply_chat_template</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">messages</span></span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">tokenize</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-language syntax-python\">False</span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">add_generation_prompt</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-language syntax-python\">True</span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span>\n                                          <span class=\"syntax-variable syntax-parameter syntax-python\">enable_thinking</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-language syntax-python\">True</span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-variable syntax-parameter syntax-python\">truncate_history_thinking</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span><span class=\"syntax-constant syntax-language syntax-python\">False</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span>\n</span></span></code></pre>\n<h3>Parsing Thinking vs. Final Answer</h3>\n<pre><code class=\"language-python\"><span class=\"code-snippet\" data-language=\"Python\"><span class=\"syntax-source syntax-python\"><span class=\"syntax-meta syntax-statement syntax-import syntax-python\"><span class=\"syntax-keyword syntax-control syntax-import syntax-python\">import</span> <span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">re</span></span></span>\n\n<span class=\"syntax-meta syntax-function syntax-python\"><span class=\"syntax-storage syntax-type syntax-function syntax-python\">def</span> <span class=\"syntax-entity syntax-name syntax-function syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">parse_model_output</span></span></span><span class=\"syntax-meta syntax-function syntax-parameters syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-parameters syntax-begin syntax-python\">(</span></span><span class=\"syntax-meta syntax-function syntax-parameters syntax-python\"><span class=\"syntax-variable syntax-parameter syntax-python\">text</span><span class=\"syntax-punctuation syntax-section syntax-parameters syntax-end syntax-python\">)</span></span><span class=\"syntax-meta syntax-function syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-function syntax-begin syntax-python\">:</span></span>\n    <span class=\"syntax-comment syntax-block syntax-documentation syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-comment syntax-begin syntax-python\">&quot;&quot;&quot;</span>Separate thinking content from final answer.<span class=\"syntax-punctuation syntax-definition syntax-comment syntax-end syntax-python\">&quot;&quot;&quot;</span></span>\n    <span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">think_match</span></span> <span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span> <span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">re</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span></span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">search</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-storage syntax-type syntax-string syntax-python\">r</span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-single syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&#39;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-single syntax-python\"><span class=\"syntax-source syntax-regexp syntax-python\">&lt;think&gt;<span class=\"syntax-meta syntax-group syntax-regexp\"><span class=\"syntax-punctuation syntax-definition syntax-group syntax-begin syntax-regexp\">(</span><span class=\"syntax-constant syntax-character syntax-character-class syntax-regexp\">.</span><span class=\"syntax-keyword syntax-operator syntax-quantifier syntax-regexp\">*?</span><span class=\"syntax-punctuation syntax-definition syntax-group syntax-end syntax-regexp\">)</span></span>&lt;/think&gt;</span><span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&#39;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">text</span></span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">re</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span><span class=\"syntax-variable syntax-other syntax-constant syntax-python\">DOTALL</span></span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span>\n    <span class=\"syntax-meta syntax-statement syntax-if syntax-python\"><span class=\"syntax-keyword syntax-control syntax-flow syntax-conditional syntax-python\">if</span> <span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">think_match</span></span><span class=\"syntax-punctuation syntax-section syntax-block syntax-conditional syntax-python\">:</span></span>\n        <span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">thinking</span></span> <span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span> <span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">think_match</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span></span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">group</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-constant syntax-numeric syntax-integer syntax-decimal syntax-python\">1</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span><span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span></span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">strip</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span>\n        <span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">answer_start</span></span> <span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span> <span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">text</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span></span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">find</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-single syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&#39;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-single syntax-python\">&lt;/think&gt;<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&#39;</span></span></span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span> <span class=\"syntax-keyword syntax-operator syntax-arithmetic syntax-python\">+</span> <span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-support syntax-function syntax-builtin syntax-python\">len</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-single syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&#39;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-single syntax-python\">&lt;/think&gt;<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&#39;</span></span></span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span>\n        <span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">answer_end</span></span> <span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span> <span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">text</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span></span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">find</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-single syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&#39;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-single syntax-python\">&lt;|im_end|&gt;<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&#39;</span></span></span><span class=\"syntax-punctuation syntax-separator syntax-arguments syntax-python\">,</span> <span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">answer_start</span></span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span>\n        <span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">answer</span></span> <span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span> <span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">text</span></span></span><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-brackets syntax-begin syntax-python\">[</span></span><span class=\"syntax-meta syntax-item-access syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">answer_start</span></span><span class=\"syntax-punctuation syntax-separator syntax-slice syntax-python\">:</span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">answer_end</span></span></span><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-brackets syntax-end syntax-python\">]</span></span><span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span></span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">strip</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span> <span class=\"syntax-meta syntax-statement syntax-if syntax-python\"><span class=\"syntax-keyword syntax-control syntax-flow syntax-conditional syntax-python\">if</span> <span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">answer_end</span></span> <span class=\"syntax-keyword syntax-operator syntax-comparison syntax-python\">!=</span> <span class=\"syntax-keyword syntax-operator syntax-arithmetic syntax-python\">-</span><span class=\"syntax-constant syntax-numeric syntax-integer syntax-decimal syntax-python\">1</span> <span class=\"syntax-keyword syntax-control syntax-flow syntax-else syntax-inline syntax-python\">else</span> <span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">text</span></span></span><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-brackets syntax-begin syntax-python\">[</span></span><span class=\"syntax-meta syntax-item-access syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">answer_start</span></span><span class=\"syntax-punctuation syntax-separator syntax-slice syntax-python\">:</span></span><span class=\"syntax-meta syntax-item-access syntax-python\"><span class=\"syntax-punctuation syntax-section syntax-brackets syntax-end syntax-python\">]</span></span><span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span></span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">strip</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span></span>\n    <span class=\"syntax-meta syntax-statement syntax-conditional syntax-python\"><span class=\"syntax-keyword syntax-control syntax-flow syntax-conditional syntax-python\">else</span><span class=\"syntax-punctuation syntax-section syntax-block syntax-python\">:</span></span>\n        <span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">thinking</span></span>, <span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">answer</span></span> <span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span> <span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-python\">&quot;</span></span></span><span class=\"syntax-meta syntax-string syntax-python\"><span class=\"syntax-string syntax-quoted syntax-double syntax-python\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-python\">&quot;</span></span></span>, <span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">text</span><span class=\"syntax-punctuation syntax-accessor syntax-dot syntax-python\">.</span></span><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">strip</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span>\n    <span class=\"syntax-keyword syntax-control syntax-flow syntax-return syntax-python\">return</span> <span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">thinking</span></span>, <span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">answer</span></span>\n\n<span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">thinking</span></span>, <span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">answer</span></span> <span class=\"syntax-keyword syntax-operator syntax-assignment syntax-python\">=</span> <span class=\"syntax-meta syntax-function-call syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-variable syntax-function syntax-python\">parse_model_output</span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-begin syntax-python\">(</span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-python\"><span class=\"syntax-meta syntax-qualified-name syntax-python\"><span class=\"syntax-meta syntax-generic-name syntax-python\">output_text</span></span></span><span class=\"syntax-punctuation syntax-section syntax-arguments syntax-end syntax-python\">)</span></span>\n</span></span></code></pre>\n<hr />\n<h2>Using with Agentic Coding Harnesses</h2>\n<p>Granite models can serve as the backbone for agentic coding tools. Because they support reasoning and tool calling through the OpenAI-compatible API, they integrate with popular agentic harnesses without extra adapters. Start the vLLM server, then follow the harness-specific instructions below.</p>\n<h3>OpenCode</h3>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://opencode.ai\">OpenCode</a> is an AI coding agent that runs in your terminal.</p>\n<p><strong>Install:</strong></p>\n<pre><code class=\"language-bash\"><span class=\"code-snippet\" data-language=\"Shell\"><span class=\"syntax-source syntax-shell syntax-bash\"><span class=\"syntax-meta syntax-function-call syntax-shell\"><span class=\"syntax-variable syntax-function syntax-shell\">curl</span></span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-shell\"><span class=\"syntax-variable syntax-parameter syntax-option syntax-shell\"><span class=\"syntax-punctuation syntax-definition syntax-parameter syntax-shell\"> -</span>fsSL</span> https://opencode.ai/install</span> <span class=\"syntax-keyword syntax-operator syntax-logical syntax-pipe syntax-shell\">|</span> <span class=\"syntax-meta syntax-function-call syntax-shell\"><span class=\"syntax-variable syntax-function syntax-shell\">bash</span></span>\n</span></span></code></pre>\n<p><strong>Configure</strong> <code>~/.config/opencode/opencode.json</code>:</p>\n<pre><code class=\"language-json\"><span class=\"code-snippet\" data-language=\"JSON\"><span class=\"syntax-source syntax-json\"><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-json\"><span class=\"syntax-punctuation syntax-section syntax-dictionary syntax-begin syntax-json\">{</span>\n  <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>$schema<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>https://opencode.ai/config.json<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-pair syntax-json\">,</span></span>\n  <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>model<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>local/granite-4.2-30b<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-pair syntax-json\">,</span></span>\n  <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>provider<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-json\"><span class=\"syntax-punctuation syntax-section syntax-dictionary syntax-begin syntax-json\">{</span>\n    <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>local<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-json\"><span class=\"syntax-punctuation syntax-section syntax-dictionary syntax-begin syntax-json\">{</span>\n      <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>npm<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>@ai-sdk/openai-compatible<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-pair syntax-json\">,</span></span>\n      <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>name<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>vLLM (local)<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-pair syntax-json\">,</span></span>\n      <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>options<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-json\"><span class=\"syntax-punctuation syntax-section syntax-dictionary syntax-begin syntax-json\">{</span>\n        <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>baseURL<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>http://localhost:8000/v1<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-pair syntax-json\">,</span></span>\n        <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>apiKey<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>EMPTY<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span>\n      </span><span class=\"syntax-punctuation syntax-section syntax-dictionary syntax-end syntax-json\">}</span></span><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-pair syntax-json\">,</span></span>\n      <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>models<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-json\"><span class=\"syntax-punctuation syntax-section syntax-dictionary syntax-begin syntax-json\">{</span>\n        <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>granite-4.2-30b<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-json\"><span class=\"syntax-punctuation syntax-section syntax-dictionary syntax-begin syntax-json\">{</span>\n          <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>name<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>Granite 4.2 30B<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-pair syntax-json\">,</span></span>\n          <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>limit<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-json\"><span class=\"syntax-punctuation syntax-section syntax-dictionary syntax-begin syntax-json\">{</span>\n            <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>context<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-constant syntax-numeric syntax-json\">131072</span><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-pair syntax-json\">,</span></span>\n            <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>output<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-constant syntax-numeric syntax-json\">8192</span>\n          </span><span class=\"syntax-punctuation syntax-section syntax-dictionary syntax-end syntax-json\">}</span></span>\n        </span><span class=\"syntax-punctuation syntax-section syntax-dictionary syntax-end syntax-json\">}</span></span>\n      </span><span class=\"syntax-punctuation syntax-section syntax-dictionary syntax-end syntax-json\">}</span></span>\n    </span><span class=\"syntax-punctuation syntax-section syntax-dictionary syntax-end syntax-json\">}</span></span>\n  </span><span class=\"syntax-punctuation syntax-section syntax-dictionary syntax-end syntax-json\">}</span></span>\n</span><span class=\"syntax-punctuation syntax-section syntax-dictionary syntax-end syntax-json\">}</span></span>\n</span></span></code></pre>\n<p><strong>Run:</strong></p>\n<pre><code class=\"language-bash\"><span class=\"code-snippet\" data-language=\"Shell\"><span class=\"syntax-source syntax-shell syntax-bash\"><span class=\"syntax-meta syntax-function-call syntax-shell\"><span class=\"syntax-variable syntax-function syntax-shell\">opencode</span></span>\n<span class=\"syntax-meta syntax-function-call syntax-shell\"><span class=\"syntax-variable syntax-function syntax-shell\">opencode</span></span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-shell\"> run <span class=\"syntax-string syntax-quoted syntax-double syntax-shell\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-shell\">&quot;</span>your task description<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-shell\">&quot;</span></span></span>\n</span></span></code></pre>\n<p>For full documentation, see <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://opencode.ai/docs\">opencode.ai/docs</a>.</p>\n<h3>Pi</h3>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://pi.dev\">Pi</a> is a minimal agent harness for AI-powered coding that runs in your terminal. It supports custom providers via a <code>models.json</code> configuration file.</p>\n<p><strong>Install:</strong></p>\n<pre><code class=\"language-bash\"><span class=\"code-snippet\" data-language=\"Shell\"><span class=\"syntax-source syntax-shell syntax-bash\"><span class=\"syntax-meta syntax-function-call syntax-shell\"><span class=\"syntax-variable syntax-function syntax-shell\">curl</span></span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-shell\"><span class=\"syntax-variable syntax-parameter syntax-option syntax-shell\"><span class=\"syntax-punctuation syntax-definition syntax-parameter syntax-shell\"> -</span>fsSL</span> https://pi.dev/install.sh</span> <span class=\"syntax-keyword syntax-operator syntax-logical syntax-pipe syntax-shell\">|</span> <span class=\"syntax-meta syntax-function-call syntax-shell\"><span class=\"syntax-variable syntax-function syntax-shell\">sh</span></span>\n</span></span></code></pre>\n<p><strong>Configure</strong> <code>~/.pi/agent/models.json</code>:</p>\n<pre><code class=\"language-json\"><span class=\"code-snippet\" data-language=\"JSON\"><span class=\"syntax-source syntax-json\"><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-json\"><span class=\"syntax-punctuation syntax-section syntax-dictionary syntax-begin syntax-json\">{</span>\n  <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>providers<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-json\"><span class=\"syntax-punctuation syntax-section syntax-dictionary syntax-begin syntax-json\">{</span>\n    <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>vllm<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-json\"><span class=\"syntax-punctuation syntax-section syntax-dictionary syntax-begin syntax-json\">{</span>\n      <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>baseUrl<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>http://localhost:8000/v1<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-pair syntax-json\">,</span></span>\n      <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>api<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>openai-completions<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-pair syntax-json\">,</span></span>\n      <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>apiKey<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>EMPTY<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-pair syntax-json\">,</span></span>\n      <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>compat<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-json\"><span class=\"syntax-punctuation syntax-section syntax-dictionary syntax-begin syntax-json\">{</span>\n        <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>supportsDeveloperRole<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-constant syntax-language syntax-json\">false</span><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-pair syntax-json\">,</span></span>\n        <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>supportsReasoningEffort<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-constant syntax-language syntax-json\">false</span>\n      </span><span class=\"syntax-punctuation syntax-section syntax-dictionary syntax-end syntax-json\">}</span></span><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-pair syntax-json\">,</span></span>\n      <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>models<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-meta syntax-structure syntax-array syntax-json\"><span class=\"syntax-punctuation syntax-section syntax-array syntax-begin syntax-json\">[</span>\n        <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-json\"><span class=\"syntax-punctuation syntax-section syntax-dictionary syntax-begin syntax-json\">{</span>\n          <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>id<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>granite-4.2-30b<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-pair syntax-json\">,</span></span>\n          <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>name<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>Granite 4.2 30B<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-pair syntax-json\">,</span></span>\n          <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>reasoning<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-constant syntax-language syntax-json\">true</span><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-pair syntax-json\">,</span></span>\n          <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>input<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-meta syntax-structure syntax-array syntax-json\"><span class=\"syntax-punctuation syntax-section syntax-array syntax-begin syntax-json\">[</span><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>text<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span><span class=\"syntax-punctuation syntax-section syntax-array syntax-end syntax-json\">]</span></span><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-pair syntax-json\">,</span></span>\n          <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>contextWindow<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-constant syntax-numeric syntax-json\">131072</span><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-pair syntax-json\">,</span></span>\n          <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>maxTokens<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-constant syntax-numeric syntax-json\">8192</span><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-pair syntax-json\">,</span></span>\n          <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>samplingParams<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-json\"><span class=\"syntax-punctuation syntax-section syntax-dictionary syntax-begin syntax-json\">{</span>\n            <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>temperature<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-constant syntax-numeric syntax-json\">1.0</span><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-pair syntax-json\">,</span></span>\n            <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>top_p<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-constant syntax-numeric syntax-json\">0.95</span>\n          </span><span class=\"syntax-punctuation syntax-section syntax-dictionary syntax-end syntax-json\">}</span></span><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-pair syntax-json\">,</span></span>\n          <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>cost<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-json\"><span class=\"syntax-punctuation syntax-section syntax-dictionary syntax-begin syntax-json\">{</span> <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>input<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-constant syntax-numeric syntax-json\">0</span><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-pair syntax-json\">,</span></span> <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>output<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-constant syntax-numeric syntax-json\">0</span><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-pair syntax-json\">,</span></span> <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>cacheRead<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-constant syntax-numeric syntax-json\">0</span><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-pair syntax-json\">,</span></span> <span class=\"syntax-meta syntax-structure syntax-dictionary syntax-key syntax-json\"><span class=\"syntax-string syntax-quoted syntax-double syntax-json\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-json\">&quot;</span>cacheWrite<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-json\">&quot;</span></span></span><span class=\"syntax-meta syntax-structure syntax-dictionary syntax-value syntax-json\"><span class=\"syntax-punctuation syntax-separator syntax-dictionary syntax-key-value syntax-json\">:</span> <span class=\"syntax-constant syntax-numeric syntax-json\">0</span> </span><span class=\"syntax-punctuation syntax-section syntax-dictionary syntax-end syntax-json\">}</span></span>\n        </span><span class=\"syntax-punctuation syntax-section syntax-dictionary syntax-end syntax-json\">}</span></span>\n      <span class=\"syntax-punctuation syntax-section syntax-array syntax-end syntax-json\">]</span></span>\n    </span><span class=\"syntax-punctuation syntax-section syntax-dictionary syntax-end syntax-json\">}</span></span>\n  </span><span class=\"syntax-punctuation syntax-section syntax-dictionary syntax-end syntax-json\">}</span></span>\n</span><span class=\"syntax-punctuation syntax-section syntax-dictionary syntax-end syntax-json\">}</span></span>\n</span></span></code></pre>\n<p><strong>Run:</strong></p>\n<pre><code class=\"language-bash\"><span class=\"code-snippet\" data-language=\"Shell\"><span class=\"syntax-source syntax-shell syntax-bash\"><span class=\"syntax-meta syntax-function-call syntax-shell\"><span class=\"syntax-variable syntax-function syntax-shell\">pi</span></span>\n</span></span></code></pre>\n<p>Then select the <code>granite-4.2-30b</code> model with <code>/model</code> or <code>Ctrl+L</code> in the interactive session.</p>\n<p>For full documentation, see <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://pi.dev/docs/latest\">pi.dev/docs</a>.</p>\n<h3>OpenHands</h3>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.openhands.dev\">OpenHands</a> is an AI software engineer that can plan, write code, and execute commands.</p>\n<ol>\n<li>\n<p><strong>Install and launch OpenHands</strong> following the <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://docs.openhands.dev/openhands/usage/agent-canvas/setup\">official installation guide</a>.</p>\n</li>\n<li>\n<p><strong>Configure the LLM</strong> in the OpenHands settings with:</p>\n<ul>\n<li><strong>Model:</strong> <code>granite-4.2-30b</code></li>\n<li><strong>Base URL:</strong> <code>http://localhost:8000/v1</code></li>\n<li><strong>API Key:</strong> your vLLM <code>--api-key</code> value</li>\n</ul>\n</li>\n</ol>\n<blockquote>\n<p><strong>Note:</strong> The <code>openai/</code> prefix is required when connecting to OpenAI-compatible endpoints like vLLM. Refer to the <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://docs.openhands.dev/openhands/usage/llms/local-llms\">OpenHands local LLM documentation</a> for detailed setup instructions, troubleshooting, and alternative installation methods.</p>\n</blockquote>\n<hr />\n<p><strong>Resources:</strong></p>\n<ul>\n<li><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://huggingface.co/collections/ibm-granite/granite-42-language-models\">Granite 4.2 HF Collection</a></li>\n<li><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://github.com/ibm-granite/granite-4.2-language-models\">GitHub: ibm-granite/granite-4.2-language-models</a></li>\n<li><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://www.ibm.com/granite/docs/\">Granite Documentation</a></li>\n<li><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://github.com/ibm-granite-community/\">Granite Community Resources</a></li>\n<li><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://research.ibm.com/blog/code-alchemy-for-synthetic-code\">Code Alchemy</a></li>\n</ul>\n",
  "content_markdown": "*A technical walkthrough of how we built the Granite 4.2 reasoning model family.*\n\n**Authors:** Granite Team, IBM\n\n* * *\n\n**TL;DR:** Granite 4.2 is our first family of dense, decoder-only reasoning LLMs, released in three sizes: **3B, 8B, and 30B**. These models are post-trained from [Granite-4.1](https://huggingface.co/collections/ibm-granite/granite-41-language-models) base models. [Granite-4.1](https://huggingface.co/collections/ibm-granite/granite-41-language-models) base models were pre-trained from scratch on roughly 15T tokens with a five-phase strategy that extends the context window to 512K tokens, supervised fine-tuned on chain-of-thought, reasoning, and agentic-trajectory data, then post-trained with a **multi-stage reinforcement learning pipeline**. That pipeline includes agentic RL, where the 8B and 30B models learn to act with tools inside real sandboxed environments. Every model has a **thinking / non-thinking** switch, a **low-effort** thinking mode that spends a short reasoning budget on easy questions, and native tool calling. All Granite 4.2 models are released under the Apache 2.0 license.\n\n**Links:**\n\n- [Granite 4.2 HF Collection](https://huggingface.co/collections/ibm-granite/granite-42-language-models)\n- [GitHub Repository](https://github.com/ibm-granite/granite-4.2-language-models)\n- [Granite Docs](https://www.ibm.com/granite/docs/)\n\n* * *\n\n## Overview\n\nGranite 4.2 is the reasoning-focused release of the Granite language-model family. Earlier Granite releases were strong instruction-following assistants; Granite 4.2 adds explicit reasoning. Every model can produce a chain of thought before its answer and can run in **thinking** or **non-thinking** mode depending on how much deliberation a task needs. A **low-effort** mode falls between the two, spending a short reasoning budget on easy questions.\n\nThe three sizes (**3B, 8B, and 30B**) share the same architectural design and follow the same training pipeline (base model, SFT, then multi-stage RL), each at its own scale. All three are strong reasoners and instruction followers. The clearest capability split shows up in post-training. The **8B and 30B** models additionally go through an **agentic RL** block that teaches them to operate as agents: calling tools, editing and running code, driving a terminal, and searching the web inside real environments. Every model supports native tool calling. Served through an OpenAI-compatible endpoint (for example, with vLLM), it emits tool calls in the OpenAI function-calling format and plugs into agentic harnesses without extra glue. Granite 4.2 is also supported in SGLang, see the [SGLang cookbook](https://docs.sglang.io/cookbook/autoregressive/IBM/Granite-4.2) for a ready-to-serve recipe.\n\nThe rest of this post walks through the build: architecture, pre-training, supervised fine-tuning, the multi-stage RL pipeline, and results.\n\n* * *\n\n## Model Architecture\n\nGranite 4.2 models are built on a decoder-only dense transformer architecture with the following core components:\n\n- **Attention:** Grouped Query Attention (GQA) with 40 attention heads and 8 KV heads\n- **Position Embedding:** Rotary Position Embedding (RoPE) with θ = 10,000,000\n- **Feed-Forward:** MLP with SwiGLU activation\n- **Normalization:** RMSNorm (ε = 1e-5)\n- **Embeddings:** Separate input/output embeddings (not tied)\n- **Precision:** bfloat16\n\n| Component                 | 3B Dense | 8B Dense | 30B Dense |\n| ------------------------- | -------- | -------- | --------- |\n| Embedding size            | 2560     | 4096     | 4096      |\n| Number of layers          | 40       | 40       | 64        |\n| Attention head size       | 64       | 128      | 128       |\n| Number of attention heads | 40       | 32       | 32        |\n| Number of KV heads        | 8        | 8        | 8         |\n| MLP hidden size           | 8192     | 12800    | 32768     |\n| MLP activation            | SwiGLU   | SwiGLU   | SwiGLU    |\n| Sequence length           | 131072   | 131072   | 131072    |\n| Position embedding        | RoPE     | RoPE     | RoPE      |\n| \\# Parameters             | 3B       | 8B       | 30B       |\n\n* * *\n\n## Pre-Training\n\nGranite 4.2 models are post-trained from [Granite-4.1](https://huggingface.co/collections/ibm-granite/granite-41-language-models) base models. [Granite-4.1](https://huggingface.co/collections/ibm-granite/granite-41-language-models) base models were trained from scratch on approximately **15 trillion tokens** using a five-phase training strategy. Phases 1–2 focus on foundational pre-training, phases 3–4 perform mid-training with progressively higher-quality data annealing, and phase 5 introduces long-context training, extending the context window to **512K tokens**. Each phase uses a distinct data mixture and learning-rate schedule, gradually shifting from broad web-scale data toward more curated, high-quality sources.\n\nThe pre-training recipe closely follows the previous generation; for a detailed treatment of the data blend, phase schedule, and long-context extension, see the [Granite 4.1 blog](https://huggingface.co/blog/ibm-granite/granite-4-1).\n\n* * *\n\n## SFT: Data Preparation & Quality Control\n\nSupervised fine-tuning (SFT) turns the base model into a reliable instruction-following, reasoning, and tool-using assistant. The SFT data mixture combines agentic (31.6%) and non-agentic (68.4%) data, totaling approximately 7.2 million samples, or roughly 100B tokens, of which about 65B are trainable.\n\nThe **agentic corpus** covers a broad range of domains, including software engineering (SWE, 69%), tool calling (12.1%), terminal use (8.0%), math (3.5%), search (0.8%), and action (0.2%). These samples and trajectories are generated using a diverse set of agent scaffolds and harnesses, including OpenHands, OpenCode, Terminus-2, SWE-agent, OpenResearcher, MiniSWE, OpenSeeker, EnvScaler, Gemini CLI, Hermes, Codex, and Goose. The agentic data combines samples from both open-source datasets and our own synthetically generated RL environments, spanning a variety of agent–harness combinations.\n\nThe **non-agentic corpus** consists of several major categories: instruction following (18.8%), coding (18.8%), math (14.6%), multilingual (7.0%), science (5.4%), reasoning (3.0%), and safety (0.8%).\n\n### Data Quality Control\n\nWe apply multiple stages of quality control before a sample enters the final SFT mixture. First, data from different sources is normalized and reformatted into a consistent OpenAI Chat format, making the conversation structure and tool interactions uniform across datasets and scaffolds.\n\nWe then use GPT-OSS-120B and Gemma 4 as LLM-based judges to assess sample quality. Low-scoring samples are removed, as are samples containing hallucinated or fabricated information, invalid tool interactions, or tool calls to functions that are not defined in the corresponding tool list. Several targeted, dataset-specific heuristic rules are also applied where appropriate to further improve quality and remove known sources of noise.\n\nFinally, we perform both local and global deduplication. Deduplication is based on SHA-256 hashes computed over the combination of the `tools` and `messages` fields, removing duplicate samples both within individual data sources and across the overall SFT mixture.\n\n### SFT Training Details\n\nThe complete corpus is first globally shuffled to reduce ordering effects and ensure that samples from different domains are well mixed during training. The shuffled corpus is then partitioned into equally sized `.parquet` shards, which are tokenized using the model's tokenizer and chat template and prepared for large-scale distributed training.\n\nBefore launching the final large-scale runs, we tune hyperparameters on representative configurations, sweeping learning-rate schedules, initial learning rates, and warm-up ratios to find settings that train stably across model sizes. The final training configuration is summarized below:\n\n| Parameter                | Value                                                 |\n| ------------------------ | ----------------------------------------------------- |\n| Compute                  | 32–128 nodes (by model size), 4× Grace/GB200 per node |\n| Sequence length (packed) | 131,072 (128K)                                        |\n| Global batch size        | 128                                                   |\n| Learning rate            | 1.0e-5, constant after warm-up; 3.0e-6 for Phase 2    |\n| LR warm-up               | 2.5% of `train_iters`                                 |\n| Training duration        | \\~2 epochs                                            |\n| Parallelism              | TP=2, PP=1, CP=4 or CP=2                              |\n\n### Phase 2 SFT for the 30B Model\n\nFor the 30B model, we additionally perform a second phase of SFT focused specifically on agentic coding. In this phase, agentic, SWE, and coding data are upsampled to increase their effective contribution to the training distribution, while approximately 16% of the mixture is retained as replay data from the original SFT corpus.\n\nThe 30B model is then fine-tuned for roughly one additional epoch at a lower learning rate of 3.0e-6. This targeted second phase increases the model's exposure to agentic coding trajectories without discarding the capabilities acquired during the initial SFT phase.\n\n* * *\n\n## Reinforcement Learning: A Multi-Stage, Multi-Environment Pipeline\n\nAfter SFT, we apply a **multi-stage, multi-environment reinforcement learning pipeline**. Rather than a single RL pass, we run a *chain* of focused stages spanning many environments: math, code, science, instruction following, tool use, and structured output, then software engineering, terminal use, and web search. Each stage is an independent RL run that targets one capability and warm-starts from the previous stage's checkpoint.\n\n[![Granite 4.2 staged RL curriculum](https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/TuxEn5hma0bQf3-z_Ab6c.png)](https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/TuxEn5hma0bQf3-z_Ab6c.png)\n\n*Figure 1. The staged RL curriculum. Foundational RL (verifiable rewards + skill boosters) runs for all sizes; the agentic RL block (SWE → Terminal → Search) runs for 8B and 30B only. Every model finishes with RLHF. Each stage is a separate GRPO run that warm-starts from the previous checkpoint.*\n\n### Training Methodology\n\nEvery stage trains with **asynchronous GRPO** ([Group Relative Policy Optimization](https://arxiv.org/abs/2402.03300)), so the generator and trainer halves of the loop never block on each other. A pool of generation workers keeps sampling responses and dropping the finished trajectories into a shared buffer; once the buffer holds a full step's worth, the trainer pulls that batch, takes an optimizer step, and streams the updated parameters back to the generator workers without pausing them. A refresh can land partway through a rollout, leaving a single trajectory stitched together from two adjacent policy versions. We allow this instead of paying to prevent it: the workers reuse their existing KV cache rather than rebuilding it after each refresh, and the one guardrail is a limit that keeps them from drifting more than a single update behind the trainer, which bounds how off-policy any sample can get. Whatever mismatch survives that limit is handled in the objective by *truncated importance sampling*, which clamps the train-versus-generation log-probability ratio to a fixed ceiling so a handful of stale tokens cannot dominate an update.\n\nAdvantages are group-relative with a **leave-one-out baseline**: each response is judged against the mean reward of the *other* samples drawn for the same prompt, which removes the need for a separate value network. To make this concrete, take **RLVR**, the first and longest-running stage: each step pairs **256 prompts** with **16 sampled responses apiece** for a **4,096**-example batch, which the trainer consumes in a single optimizer step before the next rollout begins. Later stages keep this machinery unchanged and adjust only the per-stage shape, shown next.\n\n#### RL training configuration\n\nThe pipeline keeps a common backbone of hyperparameters across every stage, which makes the curriculum easier to run and compare. A handful of knobs are fixed everywhere:\n\n| Parameter              | Value (shared across stages)                              |\n| ---------------------- | --------------------------------------------------------- |\n| Algorithm              | GRPO (no value network; group-relative advantages)        |\n| Training stack         | NeMo-RL (Megatron-Core + vLLM) with NeMo-Gym environments |\n| Ratio clip (min / max) | 0.2 / 0.28                                                |\n| Micro-batch size       | 1                                                         |\n| Parallelism            | tensor-parallel 2–4; no pipeline- or context-parallelism  |\n\nWhat changes from stage to stage is the *shape* of each run: how many prompts and generations per step, how long the context is, whether the agent loop runs, and how hard we pull back toward the reference policy. The table below gives the exact settings for the **30B** chain, stage by stage:\n\n| Stage        | Prompts/step | Gens/prompt | Max seq len | Rollout turns | KL   | LR       |\n| ------------ | ------------ | ----------- | ----------- | ------------- | ---- | -------- |\n| RLVR (×3)    | 256          | 16          | 64K         | 1             | 0    | 5e-7     |\n| IF booster   | 256          | 16          | 64K         | 1             | 0    | 5e-7     |\n| Code booster | 64           | 16          | 64K         | 1             | 0.05 | 5e-7     |\n| SWE 1        | 64           | 16          | 128K        | 1             | 0.01 | 5e-7     |\n| SWE 2        | 32           | 16          | 128K        | **128**       | 0    | 5e-7     |\n| Terminal     | 8            | 32          | 64K         | **64**        | 0.01 | **1e-6** |\n| Search       | 32           | 16          | 128K        | **64**        | 0.01 | 5e-7     |\n| RLHF         | 128          | 16          | 48K         | 1             | 0.05 | 5e-7     |\n\n*Parameters shown for the 30B model. Global batch size = prompts/step × generations/prompt (e.g. 256 × 16 = 4096 for RLVR). The 3B and 8B models use the same recipe and hyperparameters with **fewer stages** (see [How the Three Sizes Differ](https://huggingface.co/blog/ibm-granite/granite-4-2#how-the-three-sizes-differ)); the stage list is what changes, not the knobs.*\n\nThe **KL schedule** follows the reward type: explore freely where the reward is objective and verifiable (RLVR and SWE 2 run at KL 0), and stay close to the reference where the objective is preference, safety, or a narrow skill graft (RLHF and the code booster use KL 0.05). The **rollout-turns** column counts the environment interactions *GRPO itself* sees per rollout. In every case the model is trained on complete, real-environment trajectories.\n\n### The Staged Curriculum\n\nEach stage is a separate RL run with a single objective and its own reward signal. When it finishes, its policy is exported to Hugging Face format and becomes the base model for the next stage, so the pipeline is a sequence of warm-starts:\n\n```\nSFT ─▶ RLVR ─▶ Skill boosters ─▶ SWE agent ─▶ Terminal ─▶ Search ─▶ RLHF\n      └──────── foundational RL ────────┘   └──────── agentic RL (8B / 30B) ────────┘\n```\n\nThe **8B and 30B** models follow the full ladder. The **3B** model takes a shortened path: foundational RL and alignment, without the agentic block.\n\n### Reward Signals\n\nA stage is defined mostly by *how it is rewarded*. Across the pipeline there are three reward types, and a single stage can use more than one:\n\n| Reward type                  | What it measures                                                              | Used in                 |\n| ---------------------------- | ----------------------------------------------------------------------------- | ----------------------- |\n| **Verifiable**               | Exact-match, unit tests, format checkers, rule-based checkers on ground truth | RLVR · boosters · SWE   |\n| **Reward model / LLM judge** | Open-ended quality, preference, safety, answer correctness                    | RLVR · Search · RLHF    |\n| **Agentic outcome**          | Did the model actually solve a task in a real environment?                    | SWE · Terminal · Search |\n\nVerifiable rewards are objective and hard to game, so the pipeline front-loads them. Judge- and preference-based rewards handle open-ended qualities that no checker can express. Agentic-outcome rewards are the sparsest: often a single bit at the end of a long tool-use trajectory.\n\n### Foundational RL: Build the Skills\n\n#### RLVR: verifiable-reward RL\n\nRLVR is the foundational stage and the broadest data mix in the pipeline: a single blended dataset spanning many verifiable domains.\n\n- **Math:** chain-of-thought with boxed-answer checking, plus formal proving in **Lean**\n- **Competitive coding:** solutions checked against hidden tests in a sandbox\n- **STEM / graduate-level science MCQA** and general knowledge\n- **Instruction following:** structured-output and inverse-instruction tasks\n- **Tool / function calling:** single-step tool use\n- **Reasoning puzzles** and **abstention** (knowing when to refuse)\n\nEach task type carries its own verifier, so the reward is grounded per example. RLVR runs for **two rounds on 3B and 8B, and three on 30B**. Each round is a fresh warm-started run on a re-weighted mix of public and internally curated RL data.\n\n#### Skill boosters: targeted lifts\n\nAfter RLVR, a few short **booster** stages sharpen specific capabilities that benefit from concentrated training focusing on the following domains:\n\n- **Instruction following (IF):** multi-turn chat, inverse-IFEval, structured outputs\n- **Code:** competitive coding only\n\nBoosters are small, focused runs. A light KL penalty keeps the model close to its current behavior while nudging one skill.\n\n### Agentic RL: Learning to Act (8B / 30B)\n\nIn the agentic stages the model learns to **act**: call tools, observe results, and iterate inside a real environment, rewarded on whether the task was actually solved. These stages share the same shape: multi-turn tool use, real (not simulated) environments, sparse outcome rewards, and GRPO, warm-started from the coding-boosted checkpoint. They run in order: **SWE → Terminal → Search**.\n\n[![Agentic RL environments](https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/FxSMYasHnR4vpv263_DAL.png)](https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/FxSMYasHnR4vpv263_DAL.png)\n\n*Figure 2. The three agentic-RL environments. Each pairs a real harness with a real environment and a sparse, outcome-based reward. The 3B model runs none of these.*\n\n- **SWE agent (software engineering).** Each task is a real repository in its own sandbox. Driven by the [OpenHands](https://github.com/All-Hands-AI/OpenHands) harness, the model reads code, edits files, and runs the test suite over many internal turns. The reward is verifiable: do the hidden tests pass? Tasks are drawn from open-source SWE datasets, each instance backed by a per-repo container image.\n- **Terminal agent (terminal / OS operation).** Multi-step tasks in a live shell, run through the Harbor / [Terminus-2](https://www.harborframework.com/docs/agents/terminus-2) agent harness. The model plans a sequence of commands, observes their output, and recovers from errors. Reward is assigned when the task completed successfully. This is the one stage that drives its multi-turn agent loop at the GRPO level, with rollouts spanning up to 64 environment turns.\n- **Search agent (deep research).** The model answers hard, multi-hop questions using live web-search tool calls inside a browsing agent loop: gather evidence across hops, reason over it, and produce an answer. Because correctness here is open-ended, the reward is an LLM judge on the final answer.\n\n### Alignment: RLHF\n\nThe final stage of every model is **RLHF for human preference and safety.** It optimizes against a generative reward model (GenRM) for preference, plus a safety reward covering jailbreak resistance and appropriate refusals. This stage uses the highest KL penalty in the pipeline, aligning tone and safety without eroding the capabilities the earlier stages built. In addition to human preference and safety alignment, this stage also applies a reasoning-length penalty to discourage overly verbose reasoning behavior acquired during earlier stages.\n\n### How the Three Sizes Differ\n\nSame method and infrastructure; the difference is how far up the ladder each model goes.\n\n| Stage                          | 3B   | 8B               | 30B       |\n| ------------------------------ | ---- | ---------------- | --------- |\n| **RLVR** (verifiable)          | ×2   | ×2               | ×3        |\n| **Skill boosters**             | code | IF · GPQA · code | IF · code |\n| **SWE agent**                  | —    | ✓                | ✓         |\n| **Terminal agent**             | —    | ✓                | ✓         |\n| **Search agent**               | —    | ✓                | ✓         |\n| **RLHF** (preference + safety) | ✓    | ✓                | ✓         |\n\n3B is a strong foundational-RL model; 8B and 30B add the agentic-RL block on top, learning to act with tools in real environments.\n\n* * *\n\n## Agentic AI Infrastructure for Scalable RL\n\nReinforcement learning at this scale needs infrastructure that can drive a training loop and a fleet of live environments at the same time. This matters most in the agentic stages, where every training example is a multi-turn rollout that edits code, runs commands, or browses the web. Granite 4.2's RL runs on two open components: **[NeMo-RL](https://github.com/NVIDIA-NeMo/RL)** on the training side and **[NeMo-Gym](https://github.com/NVIDIA-NeMo/gym)** on the rollout side.\n\n[![NeMo-RL + NeMo-Gym system architecture](https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/ceRxgjK0m0kVHp1iJ7P34.png)](https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/ceRxgjK0m0kVHp1iJ7P34.png)\n\n*Figure 3. The RL system. NeMo-RL drives the GRPO loop (Megatron-Core training backend, vLLM generation, and Megatron-Bridge for HF⇄Megatron weight conversion). NeMo-Gym orchestrates rollouts and hosts the tools, sandboxes, and reward/verifier calls as pluggable **Resources**.*\n\nThe division of labor:\n\n- **NeMo-RL (training side).** [Megatron-Core](https://github.com/NVIDIA/Megatron-LM) is the training backend; [vLLM](https://github.com/vllm-project/vllm) generates rollouts; **Megatron-Bridge** converts weights between Megatron and Hugging Face formats, so each stage can export a clean HF checkpoint for the next one.\n- **NeMo-Gym (rollout side).** It exposes each environment as a set of **Resources** (verifiers, tools, sandboxes, and reward models) behind a uniform interface. This is the plug point for the agentic stages: the SWE repo sandboxes, the terminal harness, and the web-search tools all attach here, and to the training loop they look the same as a simple math verifier.\n\nThat uniformity is what makes the staged curriculum above practical: a booster's rule-based checker and a full SWE sandbox present the same interface to GRPO.\n\nThis split is also what makes the **asynchronous** training loop described above physically possible: generation and policy updates live on separate GPU pools, so the expensive generation fleet — including the live agentic environments — stays busy instead of idling through optimizer steps.\n\n* * *\n\n## Results\n\nGranite 4.2 was evaluated across agentic coding, general agentic and tool use, reasoning, chat and instruction following, and long context. The full benchmark table is below, followed by charts that break out the headline results by model size.\n\n| Task                         | 3B Dense | 8B Dense | 30B Dense |\n| ---------------------------- | -------- | -------- | --------- |\n| Agentic (Coding)             |          |          |           |\n| SWE Bench Multilingual       | NA       | 30.78    | 41.89     |\n| SWE Bench Pro                | NA       | 19.11    | 33.29     |\n| SWE Bench Verified           | NA       | 47.67    | 57.00     |\n| Terminal-Bench 2.1           | NA       | 20.56    | 29.24     |\n| Agentic (General)            |          |          |           |\n| τ³-bench                     | 45.78    | 58.06    | 62.00     |\n| BFCL (v4)                    | 52.41    | 50.29    | 61.39     |\n| ProfBench                    | 32.10    | 41.20    | 42.90     |\n| BirdBench                    | NA       | 41.07    | 41.85     |\n| GDPval                       | NA       | 1189.00  | 1225.00   |\n| Reasoning                    |          |          |           |\n| AIME25                       | 78.33    | 86.67    | 89.17     |\n| HMMT Feb25                   | 66.67    | 78.33    | 89.17     |\n| GPQA                         | 54.80    | 64.14    | 66.41     |\n| LiveCodeBench v6             | 69.71    | 73.24    | 75.77     |\n| SciCode                      | 24.11    | 36.09    | 38.76     |\n| Chat & Instruction Following |          |          |           |\n| MMLU-Pro                     | 67.84    | 74.04    | 77.60     |\n| MMLU-ProX lite (IBM)         | 27.78    | 61.06    | 66.64     |\n| Arena-Hard-V2                | 34.96    | 65.19    | 67.93     |\n| IFBench (prompt)             | 74.33    | 79.33    | 77.17     |\n| Long Context                 |          |          |           |\n| RULER 64K                    | 67.52    | 80.99    | 89.96     |\n| RULER 128K                   | 55.30    | 71.41    | 81.38     |\n\n**Supported languages:** English, German, Spanish, French, Japanese, Portuguese, Arabic, Czech, Italian, Korean, Dutch, and Chinese.\n\nThe charts below break these results out by capability area.\n\n[![Reasoning benchmarks](https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/u4p7x5-U_7Qj9w-ovsZ7D.png)](https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/u4p7x5-U_7Qj9w-ovsZ7D.png)\n\n*Figure 4. Reasoning (pass@1). Scores rise consistently with model size across math (AIME25, HMMT), science (GPQA), and code reasoning (LiveCodeBench, SciCode).*\n\n[![Agentic coding benchmarks](https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/CH5Aqnjba9hHlBteGNvw0.png)](https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/CH5Aqnjba9hHlBteGNvw0.png)\n\n*Figure 5. Agentic coding resolve rates. The agentic-RL block is trained only for 8B and 30B; the 30B model leads across SWE-Bench variants and Terminal-Bench.*\n\n[![General agentic and tool-use benchmarks](https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/mXZuw86XqEvna7Rm2OcTi.png)](https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/mXZuw86XqEvna7Rm2OcTi.png)\n\n*Figure 6. General agentic and tool-use benchmarks, reported for all three sizes.*\n\n* * *\n\n## Quantization\n\nWe also released four quantized variants of the Granite 4.2 models for inference with vLLM. The models are converted to FP8, NVFP4, and MXFP4 using LLM Compressor, and to the GGUF format using the llama.cpp framework for reduced-memory deployment.\n\n### FP8\n\nThe FP8 version is quantized with dynamic per-channel weights and per-token activations. No calibration is used.\n\n### FP4\n\nThe NVFP4 and MXFP4 versions are quantized using GPTQ calibrated on 2K samples drawn from the SFT dataset. Max context length is 2K during calibration.\n\n### GGUF\n\nConversion of the Granite 4.2 models to GGUF is done with the canonical llama.cpp tool as described in [https://github.com/IBM/gguf#gguf-conversion--quantization](https://github.com/IBM/gguf#gguf-conversion--quantization).\n\nSeveral GGUF formats are provided:\n\n- Q8\\_0\n- Q6\\_K\n- Q5\\_K\\_S\n- Q5\\_K\\_M\n- Q5\\_1\n- Q5\\_0\n- Q4\\_K\\_S\n- Q4\\_K\\_M\n- Q4\\_1\n- Q4\\_0\n- Q3\\_K\\_S\n- Q3\\_K\\_M\n- Q3\\_K\\_L\n- Q2\\_K\n\n* * *\n\n## Infrastructure\n\n### Hardware\n\nWe trained the Granite 4.2 language models on an NVIDIA GB200 NVL72 cluster hosted by CoreWeave, featuring:\n\n- A 72-GPU NVLink domain for high-speed intra-rack communication\n- A non-blocking Fat-Tree NDR 400 Gb/s InfiniBand fabric for full-bandwidth inter-rack connectivity\n- Thousands of GPUs operating at cluster scale\n\nThis infrastructure delivers the high-bandwidth, low-latency communication required for efficient large-scale distributed training.\n\n### Software Stack\n\nThe training software stack is packaged into `.sqsh` container images, each giving a run a reproducible, portable environment with its SBSA-compatible CUDA targets, Linux aarch64 Python wheels, and GPU-specific binaries pinned. The large-scale SFT runs build on an NGC PyTorch base image (Ubuntu 22.04, CUDA 12.8, Python 3.12); the RL stack runs in its own NeMo-RL container.\n\n* * *\n\n## Getting Started (Transformers)\n\n### Installation\n\n```bash\npip install torch\npip install accelerate transformers\n```\n\n### Basic Inference (Thinking Mode)\n\n```python\nimport torch\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel_path = \"ibm-granite/granite-4.2-30b\"\ntokenizer = AutoTokenizer.from_pretrained(model_path)\nmodel = AutoModelForCausalLM.from_pretrained(model_path, device_map=\"cuda\", torch_dtype=torch.bfloat16)\nmodel.eval()\n\nmessages = [\n    {\"role\": \"user\", \"content\": \"How many r's are in the word 'strawberry'?\"},\n]\n\ntext = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=True)\ninputs = tokenizer(text, return_tensors=\"pt\").to(model.device)\n\nwith torch.no_grad():\n    output = model.generate(**inputs, max_new_tokens=8192, temperature=1.0, top_p=0.95, do_sample=True)\n\nprint(tokenizer.decode(output[0][inputs.input_ids.shape[-1]:], skip_special_tokens=False))\n```\n\n**Example Output**\n\n```\n<think>\nOkay, let's see. The problem is to find how many 'r's are in the word 'strawberry'.\n\nFirst, I need to write out the word: s t r a w b e r r y.\n\nNow, I need to count the number of 'r' letters. Let's list each letter and check for 'r'.\n\n1. s – not r\n2. t – not r\n3. r – yes, that's one\n4. a – no\n5. w – no\n6. b – no\n7. e – no\n8. r – yes, that's two\n9. r – yes, that's three\n10. y – no\n\nTotal r's = 3.\n</think>\nThere are **3** r's in the word \"strawberry\".<|im_end|>\n```\n\n### Non-Thinking Mode\n\n```python\nmessages = [\n    {\"role\": \"user\", \"content\": \"What is the capital of France?\"},\n]\ntext = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)\ninputs = tokenizer(text, return_tensors=\"pt\").to(model.device)\n\noutput = model.generate(**inputs, max_new_tokens=2048, temperature=1.0, top_p=0.95, do_sample=True)\nprint(tokenizer.decode(output[0][inputs.input_ids.shape[-1]:], skip_special_tokens=False))\n```\n\n**Example Output**\n\n```\n<think></think>The capital of France is Paris.<|im_end|>\n```\n\n### Low-Effort Thinking\n\n```python\nmessages = [\n    {\"role\": \"user\", \"content\": \"What is 2 + 2?\"},\n]\ntext = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True,\n                                     enable_thinking=True, low_effort=True)\ninputs = tokenizer(text, return_tensors=\"pt\").to(model.device)\n\noutput = model.generate(**inputs, max_new_tokens=4096, temperature=1.0, top_p=0.95, do_sample=True)\nprint(tokenizer.decode(output[0][inputs.input_ids.shape[-1]:], skip_special_tokens=False))\n```\n\n**Example Output**\n\n```\n<think>\nSimple answer.\n</think>\n2 + 2 = 4.<|im_end|>\n```\n\n* * *\n\n## Tool Calling\n\nGranite models support tool calling with integrated reasoning: the model reasons about which tool to call and why before calling it. Tools are defined with the [OpenAI function definition schema](https://platform.openai.com/docs/guides/function-calling).\n\n### Basic Tool Calling\n\n```python\ntools = [\n    {\n        \"type\": \"function\",\n        \"function\": {\n            \"name\": \"get_current_weather\",\n            \"description\": \"Get the current weather for a specified city.\",\n            \"parameters\": {\n                \"type\": \"object\",\n                \"properties\": {\n                    \"city\": {\"type\": \"string\", \"description\": \"Name of the city\"}\n                },\n                \"required\": [\"city\"]\n            }\n        }\n    }\n]\n\nmessages = [\n    {\"role\": \"user\", \"content\": \"What's the weather like in Boston right now?\"},\n]\ntext = tokenizer.apply_chat_template(messages, tokenize=False, tools=tools,\n                                     add_generation_prompt=True, enable_thinking=True)\ninputs = tokenizer(text, return_tensors=\"pt\").to(model.device)\n\noutput = model.generate(**inputs, max_new_tokens=4096, temperature=1.0, top_p=0.95, do_sample=True)\nprint(tokenizer.decode(output[0][inputs.input_ids.shape[-1]:], skip_special_tokens=False))\n```\n\n**Example Output**\n\n```\n<think>\nOkay, the user is asking for the weather in Boston right now. Let me check the tools\navailable. There's a function called get_current_weather that takes a city parameter.\nI need to call that with the city set to Boston.\n</think>\n<tool_call>\n<function=get_current_weather>\n<parameter=city>\nBoston\n</parameter>\n</function>\n</tool_call>\n<|im_end|>\n```\n\n### Multi-Turn with Tool Response\n\n```python\nmessages = [\n    {\"role\": \"user\", \"content\": \"What's the weather like in Boston right now?\"},\n    {\"role\": \"assistant\", \"content\": \"<think>\\nThe user wants to know the current weather in Boston. I should call get_current_weather.\\n</think>\",\n     \"tool_calls\": [{\"function\": {\"name\": \"get_current_weather\", \"arguments\": {\"city\": \"Boston\"}}}]},\n    {\"role\": \"tool\", \"content\": '{\"temperature\": \"72°F\", \"condition\": \"Partly cloudy\", \"humidity\": \"65%\"}'},\n]\ntext = tokenizer.apply_chat_template(messages, tokenize=False, tools=tools,\n                                     add_generation_prompt=True, enable_thinking=True)\ninputs = tokenizer(text, return_tensors=\"pt\").to(model.device)\n\noutput = model.generate(**inputs, max_new_tokens=4096, temperature=1.0, top_p=0.95, do_sample=True)\nprint(tokenizer.decode(output[0][inputs.input_ids.shape[-1]:], skip_special_tokens=False))\n```\n\n**Example Output**\n\n```\n<think>\nThe tool returned the weather data for Boston: temperature 72°F, partly cloudy, humidity 65%.\nI need to present this information clearly to the user.\n</think>\nThe current weather in Boston is 72°F, partly cloudy, with 65% humidity.<|im_end|>\n```\n\n* * *\n\n## Multi-Turn Conversations\n\n### History Thinking Truncation\n\n```python\nmessages = [\n    {\"role\": \"user\", \"content\": \"What is 15 * 37?\"},\n    {\"role\": \"assistant\", \"content\": \"<think>\\nLet me calculate 15 * 37.\\n15 * 37 = 15 * 30 + 15 * 7 = 450 + 105 = 555\\n</think>\\n15 * 37 = 555\"},\n    {\"role\": \"user\", \"content\": \"Now divide that by 5\"},\n]\n\n# Default: previous thinking is stripped to save context\ntext = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True,\n                                     enable_thinking=True, truncate_history_thinking=True)\n\n# To preserve full history:\ntext_full = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True,\n                                          enable_thinking=True, truncate_history_thinking=False)\n```\n\n### Parsing Thinking vs. Final Answer\n\n```python\nimport re\n\ndef parse_model_output(text):\n    \"\"\"Separate thinking content from final answer.\"\"\"\n    think_match = re.search(r'<think>(.*?)</think>', text, re.DOTALL)\n    if think_match:\n        thinking = think_match.group(1).strip()\n        answer_start = text.find('</think>') + len('</think>')\n        answer_end = text.find('<|im_end|>', answer_start)\n        answer = text[answer_start:answer_end].strip() if answer_end != -1 else text[answer_start:].strip()\n    else:\n        thinking, answer = \"\", text.strip()\n    return thinking, answer\n\nthinking, answer = parse_model_output(output_text)\n```\n\n* * *\n\n## Using with Agentic Coding Harnesses\n\nGranite models can serve as the backbone for agentic coding tools. Because they support reasoning and tool calling through the OpenAI-compatible API, they integrate with popular agentic harnesses without extra adapters. Start the vLLM server, then follow the harness-specific instructions below.\n\n### OpenCode\n\n[OpenCode](https://opencode.ai) is an AI coding agent that runs in your terminal.\n\n**Install:**\n\n```bash\ncurl -fsSL https://opencode.ai/install | bash\n```\n\n**Configure** `~/.config/opencode/opencode.json`:\n\n```json\n{\n  \"$schema\": \"https://opencode.ai/config.json\",\n  \"model\": \"local/granite-4.2-30b\",\n  \"provider\": {\n    \"local\": {\n      \"npm\": \"@ai-sdk/openai-compatible\",\n      \"name\": \"vLLM (local)\",\n      \"options\": {\n        \"baseURL\": \"http://localhost:8000/v1\",\n        \"apiKey\": \"EMPTY\"\n      },\n      \"models\": {\n        \"granite-4.2-30b\": {\n          \"name\": \"Granite 4.2 30B\",\n          \"limit\": {\n            \"context\": 131072,\n            \"output\": 8192\n          }\n        }\n      }\n    }\n  }\n}\n```\n\n**Run:**\n\n```bash\nopencode\nopencode run \"your task description\"\n```\n\nFor full documentation, see [opencode.ai/docs](https://opencode.ai/docs).\n\n### Pi\n\n[Pi](https://pi.dev) is a minimal agent harness for AI-powered coding that runs in your terminal. It supports custom providers via a `models.json` configuration file.\n\n**Install:**\n\n```bash\ncurl -fsSL https://pi.dev/install.sh | sh\n```\n\n**Configure** `~/.pi/agent/models.json`:\n\n```json\n{\n  \"providers\": {\n    \"vllm\": {\n      \"baseUrl\": \"http://localhost:8000/v1\",\n      \"api\": \"openai-completions\",\n      \"apiKey\": \"EMPTY\",\n      \"compat\": {\n        \"supportsDeveloperRole\": false,\n        \"supportsReasoningEffort\": false\n      },\n      \"models\": [\n        {\n          \"id\": \"granite-4.2-30b\",\n          \"name\": \"Granite 4.2 30B\",\n          \"reasoning\": true,\n          \"input\": [\"text\"],\n          \"contextWindow\": 131072,\n          \"maxTokens\": 8192,\n          \"samplingParams\": {\n            \"temperature\": 1.0,\n            \"top_p\": 0.95\n          },\n          \"cost\": { \"input\": 0, \"output\": 0, \"cacheRead\": 0, \"cacheWrite\": 0 }\n        }\n      ]\n    }\n  }\n}\n```\n\n**Run:**\n\n```bash\npi\n```\n\nThen select the `granite-4.2-30b` model with `/model` or `Ctrl+L` in the interactive session.\n\nFor full documentation, see [pi.dev/docs](https://pi.dev/docs/latest).\n\n### OpenHands\n\n[OpenHands](https://www.openhands.dev) is an AI software engineer that can plan, write code, and execute commands.\n\n1. **Install and launch OpenHands** following the [official installation guide](https://docs.openhands.dev/openhands/usage/agent-canvas/setup).\n\n2. **Configure the LLM** in the OpenHands settings with:\n\n   - **Model:** `granite-4.2-30b`\n   - **Base URL:** `http://localhost:8000/v1`\n   - **API Key:** your vLLM `--api-key` value\n\n> **Note:** The `openai/` prefix is required when connecting to OpenAI-compatible endpoints like vLLM. Refer to the [OpenHands local LLM documentation](https://docs.openhands.dev/openhands/usage/llms/local-llms) for detailed setup instructions, troubleshooting, and alternative installation methods.\n\n* * *\n\n**Resources:**\n\n- [Granite 4.2 HF Collection](https://huggingface.co/collections/ibm-granite/granite-42-language-models)\n- [GitHub: ibm-granite/granite-4.2-language-models](https://github.com/ibm-granite/granite-4.2-language-models)\n- [Granite Documentation](https://www.ibm.com/granite/docs/)\n- [Granite Community Resources](https://github.com/ibm-granite-community/)\n- [Code Alchemy](https://research.ibm.com/blog/code-alchemy-for-synthetic-code)\n",
  "dateCreated": "2026-10-19T01:26:53.998049939+00:00",
  "date_created": "2026-08-25T15:14:14+00:00",
  "date_modified": "2026-08-25T15:14:14+00:00",
  "date_published": "2026-08-25T15:14:14+00:00",
  "description": "A technical walkthrough of how we built the Granite 4.2 reasoning model family. Authors: Granite Team, IBM TL;DR: Granite 4.2 is our first family of dense, decoder-only reasoning LLMs, released in three sizes: 3B, 8B, and 30B. These models…",
  "external_url": "https://huggingface.co/blog/ibm-granite/granite-4-2",
  "git": {
    "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e",
    "path": "items/huggingface-co/2026/08/2026-08-25-granite-4-2-llms-how-they-re-built.md",
    "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/huggingface-co/2026/08/2026-08-25-granite-4-2-llms-how-they-re-built.md"
  },
  "id": "urn:aggr:item:3e163091265f44c86f7ba27aec1cb050bdd930f0",
  "isBasedOn": {
    "@id": "https://huggingface.co/blog/ibm-granite/granite-4-2"
  },
  "mainEntity": {
    "@id": "https://huggingface.co/blog/ibm-granite/granite-4-2",
    "@type": "CreativeWork",
    "archivedAt": {
      "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-08-25-granite-4-2-llms-how-they-re-built/#webpage"
    },
    "dateModified": "2026-08-25T15:14:14+00:00",
    "datePublished": "2026-08-25T15:14:14+00:00",
    "headline": "Granite 4.2 LLMs: How They're Built",
    "image": {
      "@type": "ImageObject",
      "caption": null,
      "height": 90,
      "url": "https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/XHWV2L_wZsFe3ekksCgvQ.webp",
      "width": 160
    },
    "timeRequired": "PT21M",
    "url": "https://huggingface.co/blog/ibm-granite/granite-4-2",
    "wordCount": 4623
  },
  "name": "Archived snapshot: Granite 4.2 LLMs: How They're Built",
  "preview": {
    "alt": null,
    "color": null,
    "height": 90,
    "placeholder": {
      "data_url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII=",
      "hash": ""
    },
    "url": "https://cdn-uploads.huggingface.co/production/uploads/65cc2c288ebd392213e58899/XHWV2L_wZsFe3ekksCgvQ.webp",
    "width": 160
  },
  "reading_minutes": 21,
  "replicated_at": null,
  "source": "huggingface-co",
  "source_name": "Hugging Face - Blog",
  "summary": null,
  "tags": [],
  "temporalCoverage": "2026-10-19T01:26:53.998049939+00:00",
  "title": "Granite 4.2 LLMs: How They're Built",
  "url": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-08-25-granite-4-2-llms-how-they-re-built/",
  "word_count": 4623
}