{ "@context": "https://schema.org", "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2025-10-16-google-cloud-c4-brings-a-70-tco-improvement-on-gpt-oss-with/#webpage", "@type": [ "WebPage", "ArchiveComponent" ], "articleBody": "Intel and Hugging Face collaborated to demonstrate the real-world value of upgrading to Google’s latest `C4` Virtual Machine (VM) running on Intel® Xeon® 6 processors (codenamed Granite Rapids (GNR)). We specifically wanted to benchmark improvements in the text generation performance of OpenAI GPT OSS Large Language Model(LLM).\n\nThe results are in, and they are impressive, demonstrating a *1.7x* improvement in Total Cost of Ownership(TCO) over the previous-generation Google `C3` VM instances. The Google Cloud `C4` VM instance further resulted in:\n\n- 1.4x to 1.7x TPOT throughput/vCPU/dollar\n- Lower price per hour over `C3` VM\n\n## Introduction\n\nGPT OSS is a common name for an open-source Mixture of Experts (MoE) model released by OpenAI. An MoE model is a deep neural network architecture that uses specialized “expert” sub-networks and a “gating network” to decide which experts to use for a given input. MoE models allow you to scale your model capacity efficiently without linearly scaling compute costs. They also allow for specialization, where different “experts” learn different skills, allowing them to adapt to diverse data distributions.\n\nEven with very large parameters, only a small subset of experts is activated per token, making CPU inference viable.\n\nIntel and Hugging Face collaborated to merge an expert execution optimization (PR [#40304](https://github.com/huggingface/transformers/pull/40304)) to eliminate redundant computation where every expert processed all tokens to transformers. This optimization directed each expert to run only on the tokens it is routed to, removing FLOPs waste and improving utilization.\n\n![gpt_oss_expert](https://huggingface.co/datasets/Intel/blog/resolve/main/gpt-oss-on-intel-xeon/gpt_oss_expert.png)\n\n## Benchmark Scope & Hardware\n\nWe benchmarked GPT OSS under a controlled, repeatable generation workload to isolate architectural differences (GCP `C4` VMs on Intel Xeon 6 processors (GNR) vs GCP `C3` VMs on 4th Gen Intel Xeon Processors (SPR)) and MoE execution efficiency. The focus is steady‑state decoding (per‑token latency) and end‑to‑end normalized throughput with increasing batch size while keeping sequence lengths fixed. All runs use static KV cache and SDPA attention for determinism.\n\n### Configuration Summary\n\n- Model: [unsloth/gpt-oss-120b-BF16](https://huggingface.co/unsloth/gpt-oss-120b-BF16)\n- Precision: bfloat16\n- Task: Text generation\n- Input length: 1024 tokens (left‑padded)\n- Output length: 1024 tokens\n- Batch sizes: 1, 2, 4, 8, 16, 32, 64\n- Enabled features:\n - Static KV cache\n - SDPA attention backend\n- Reported metrics:\n - Throughput (Total generated tokens per second aggregated over the batch)\n\n### Hardware Under Test\n\n| Instance | Architecture | vCPUs |\n| -------- | ---------------------------------- | ----- |\n| `C3` | 4th Gen Intel Xeon processor (SPR) | 172 |\n| `C4` | Intel Xeon 6 processor (GNR) | 144 |\n\n## Create instance\n\n### `C3`\n\nVisit [`Google Cloud Console`](https://console.cloud.google.com/) and click on `create a VM` under your project. Follow the steps below to create a `176 vCPU` instance.\n\n1. pick `C3` in the `Machine configuration` and specify Machine type as `c3-standard-176`. You also need to set the `CPU platform` and turn on `all-core turbo` to make performance more stable: [![alt text](https://huggingface.co/datasets/Intel/blog/resolve/main/gpt-oss-on-intel-xeon/spr.png)](https://huggingface.co/datasets/Intel/blog/resolve/main/gpt-oss-on-intel-xeon/spr.png)\n2. configure OS and storage tab as below: [![alt text](https://huggingface.co/datasets/Intel/blog/resolve/main/gpt-oss-on-intel-xeon/spr-os.png)](https://huggingface.co/datasets/Intel/blog/resolve/main/gpt-oss-on-intel-xeon/spr-os.png)\n3. keep other configurations as default\n4. click `Create` button\n\n### `C4`\n\nVisit [`Google Cloud Console`](https://console.cloud.google.com/) and click on `create a VM` under your project. Follow the below steps to create a `144 vCPU` instance.\n\n1. pick `C4` in the `Machine configuration` tab and specify Machine type as `c4-standard-144`. You can also set the `CPU platform` and turn on all-core turbo to make performance more stable: [![alt text](https://huggingface.co/datasets/Intel/blog/resolve/main/gpt-oss-on-intel-xeon/gnr.png)](https://huggingface.co/datasets/Intel/blog/resolve/main/gpt-oss-on-intel-xeon/gnr.png)\n2. configure OS and storage tab as we need for C3.\n3. keep other configurations as default\n4. click `Create` button\n\n## Set up the environment\n\nLogin the instance with SSH and then install docker. Follow the steps below to set up the environment easily. For reproducibility, we list the versions and commits we are using in the commands.\n\n1. `$ git clone https://github.com/huggingface/transformers.git`\n2. `$ cd transformers/`\n3. `$ git checkout 26b65fb5168f324277b85c558ef8209bfceae1fe`\n4. `$ cd docker/transformers-intel-cpu/`\n5. `$ sudo docker build . -t `\n6. `$ sudo docker run -it --rm --privileged -v /home/:/workspace /bin/bash`\n\nWe are in container now, do following steps.\n\n1. `$ pip install git+https://github.com/huggingface/transformers.git@26b65fb5168f324277b85c558ef8209bfceae1fe`\n2. `$ pip install torch==2.8.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu`\n\n## Benchmark Procedure\n\nFor each batch size we\n\n1. Build a fixed-length 1024‑token left‑padded batch.\n2. Run a single warm‑up round.\n3. set `max_new_tokens=1024` and measure total latency, then get throughput=(OUTPUT\\_TOKENS∗batch\\_size)/total\\_latencythroughput = (OUTPUT\\\\\\_TOKENS \\* batch\\\\\\_size) / total\\\\\\_latency.\n\nRun `numactl -l python benchmark.py` for the following codes.\n\n```python\nimport os\nimport time\nimport torch\nfrom datasets import load_dataset\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nINPUT_TOKENS = 1024\nOUTPUT_TOKENS = 1024\n\ndef get_inputs(tokenizer, batch_size):\n dataset = load_dataset(\"ola13/small-the_pile\", split=\"train\")\n tokenizer.padding_side = \"left\"\n selected_texts = []\n for sample in dataset:\n input_ids = tokenizer(sample[\"text\"], return_tensors=\"pt\").input_ids\n if len(selected_texts) == 0 and input_ids.shape[-1] >= INPUT_TOKENS:\n selected_texts.append(sample[\"text\"])\n elif len(selected_texts) > 0:\n selected_texts.append(sample[\"text\"])\n if len(selected_texts) == batch_size:\n break\n\n return tokenizer(selected_texts, max_length=INPUT_TOKENS, padding=\"max_length\", truncation=True, return_tensors=\"pt\")\n\ndef run_generate(model, inputs, generation_config):\n inputs[\"generation_config\"] = generation_config\n model.generate(**inputs) # warm up\n pre = time.time()\n model.generate(**inputs)\n latency = (time.time() - pre)\n return latency\n\ndef benchmark(model, tokenizer, batch_size, generation_config):\n inputs = get_inputs(tokenizer, batch_size)\n generation_config.max_new_tokens = 1\n generation_config.min_new_tokens = 1\n prefill_latency = run_generate(model, inputs, generation_config)\n generation_config.max_new_tokens = OUTPUT_TOKENS\n generation_config.min_new_tokens = OUTPUT_TOKENS\n total_latency = run_generate(model, inputs, generation_config)\n decoding_latency = (total_latency - prefill_latency) / (OUTPUT_TOKENS - 1)\n throughput = OUTPUT_TOKENS * batch_size / total_latency\n\n return prefill_latency, decoding_latency, throughput\n\n\nif __name__ == \"__main__\":\n model_id = \"unsloth/gpt-oss-120b-BF16\"\n tokenizer = AutoTokenizer.from_pretrained(model_id)\n model_kwargs = {\"dtype\": torch.bfloat16}\n model = AutoModelForCausalLM.from_pretrained(model_id, **model_kwargs)\n model.config._attn_implementation=\"sdpa\"\n generation_config = model.generation_config\n generation_config.do_sample = False\n generation_config.cache_implementation=\"static\"\n\n for batch_size in [1, 2, 4, 8, 16, 32, 64]:\n print(f\"---------- Run generation with batch size = {batch_size} ----------\", flush=True)\n prefill_latency, decoding_latency, throughput = benchmark(model, tokenizer, batch_size, generation_config)\n print(f\"throughput = {throughput}\", flush=True)\n```\n\n## Results\n\n### Normalized Throughput per vCPU\n\nAcross batch sizes up to 64, Intel Xeon 6 processor‑powered `C4` consistently outperforms `C3` with a 1.4x to 1.7× throughput per-vCPU. The formula is:\n\nnormalized\\_throughput\\_per\\_vCPU=throughput\\_C4/vCPUs\\_C4throughput\\_C3/vCPUs\\_C3 \\\\text{normalized\\\\\\_throughput\\\\\\_per\\\\\\_vCPU} = \\\\frac{\\\\text{throughput\\\\\\_C4} / \\\\text{vCPUs\\\\\\_C4}} {\\\\text{throughput\\\\\\_C3} / \\\\text{vCPUs\\\\\\_C3}}\n\n![throughput-gpt-oss-per-vcpu](https://huggingface.co/datasets/Intel/blog/resolve/main/gpt-oss-on-intel-xeon/throughput-gpt-oss-per-vcpu.png)\n\n### Cost & TCO\n\nAt batch size 64, `C4` provides 1.7× the per‑vCPU throughput of `C3`; with near parity in price per vCPU (hourly cost scales linearly with vCPU count), this yields a 1.7× TCO advantage (`C3` would require 1.7× the spend for the same generated token volume).\n\nPer‑vCPU throughput ratio: throughput\\_C4/vCPUs\\_C4throughput\\_C3/vCPUs\\_C3=1.7⇒TCO\\_C3TCO\\_C4≈1.7 \\\\frac{\\\\text{throughput\\\\\\_C4} / \\\\text{vCPUs\\\\\\_C4}}{\\\\text{throughput\\\\\\_C3} / \\\\text{vCPUs\\\\\\_C3}} = 1.7 \\\\Rightarrow \\\\frac{\\\\text{TCO\\\\\\_C3}}{\\\\text{TCO\\\\\\_C4}} \\\\approx 1.7\n\n![throughput-gpt-oss-per-dollar](https://huggingface.co/datasets/Intel/blog/resolve/main/gpt-oss-on-intel-xeon/throughput-gpt-oss-per-dollar.png)\n\n## Conclusion\n\nGoogle Cloud `C4` VMs powered by Intel Xeon 6 processors (GNR) provide both impressive performance gains and better cost efficiency for large MoE inference over previous generation Google Cloud `C3` VM (powered by 4th Gen Intel Xeon processors). For GPT OSS MoE inference, we observed combined higher throughput, lower latency, and reduced cost. These results underline that thanks to targeted framework optimizations from Intel and Hugging Face, large MoE models can be efficiently served on next-generation general-purpose CPUs.\n", "authors": [], "captured_at": "2026-10-04T12:50:30.804479567+00:00", "category": "ai", "content_html": "

Intel and Hugging Face collaborated to demonstrate the real-world value of upgrading to Google’s latest C4 Virtual Machine (VM) running on Intel® Xeon® 6 processors (codenamed Granite Rapids (GNR)). We specifically wanted to benchmark improvements in the text generation performance of OpenAI GPT OSS Large Language Model(LLM).

\n

The results are in, and they are impressive, demonstrating a 1.7x improvement in Total Cost of Ownership(TCO) over the previous-generation Google C3 VM instances. The Google Cloud C4 VM instance further resulted in:

\n
    \n
  • 1.4x to 1.7x TPOT throughput/vCPU/dollar
    • \n
  • Lower price per hour over C3 VM
    • \n
\n

Introduction

\n

GPT OSS is a common name for an open-source Mixture of Experts (MoE) model released by OpenAI. An MoE model is a deep neural network architecture that uses specialized “expert” sub-networks and a “gating network” to decide which experts to use for a given input. MoE models allow you to scale your model capacity efficiently without linearly scaling compute costs. They also allow for specialization, where different “experts” learn different skills, allowing them to adapt to diverse data distributions.

\n

Even with very large parameters, only a small subset of experts is activated per token, making CPU inference viable.

\n

Intel and Hugging Face collaborated to merge an expert execution optimization (PR #40304) to eliminate redundant computation where every expert processed all tokens to transformers. This optimization directed each expert to run only on the tokens it is routed to, removing FLOPs waste and improving utilization.

\n

\"gpt_oss_expert\"

\n

Benchmark Scope & Hardware

\n

We benchmarked GPT OSS under a controlled, repeatable generation workload to isolate architectural differences (GCP C4 VMs on Intel Xeon 6 processors (GNR) vs GCP C3 VMs on 4th Gen Intel Xeon Processors (SPR)) and MoE execution efficiency. The focus is steady‑state decoding (per‑token latency) and end‑to‑end normalized throughput with increasing batch size while keeping sequence lengths fixed. All runs use static KV cache and SDPA attention for determinism.

\n

Configuration Summary

\n
    \n
  • Model: unsloth/gpt-oss-120b-BF16
    • \n
  • Precision: bfloat16
    • \n
  • Task: Text generation
    • \n
  • Input length: 1024 tokens (left‑padded)
    • \n
  • Output length: 1024 tokens
    • \n
  • Batch sizes: 1, 2, 4, 8, 16, 32, 64
    • \n
  • Enabled features:\n
      \n
    • Static KV cache
      • \n
    • SDPA attention backend
      • \n
    \n
    • \n
  • Reported metrics:\n
      \n
    • Throughput (Total generated tokens per second aggregated over the batch)
      • \n
    \n
    • \n
\n

Hardware Under Test

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
InstanceArchitecturevCPUs
C34th Gen Intel Xeon processor (SPR)172
C4Intel Xeon 6 processor (GNR)144
\n

Create instance

\n

C3

\n

Visit Google Cloud Console and click on create a VM under your project. Follow the steps below to create a 176 vCPU instance.

\n
    \n
  1. pick C3 in the Machine configuration and specify Machine type as c3-standard-176. You also need to set the CPU platform and turn on all-core turbo to make performance more stable: \"alt
    2. \n
  2. configure OS and storage tab as below: \"alt
    3. \n
  3. keep other configurations as default
    4. \n
  4. click Create button
    5. \n
\n

C4

\n

Visit Google Cloud Console and click on create a VM under your project. Follow the below steps to create a 144 vCPU instance.

\n
    \n
  1. pick C4 in the Machine configuration tab and specify Machine type as c4-standard-144. You can also set the CPU platform and turn on all-core turbo to make performance more stable: \"alt
    2. \n
  2. configure OS and storage tab as we need for C3.
    3. \n
  3. keep other configurations as default
    4. \n
  4. click Create button
    5. \n
\n

Set up the environment

\n

Login the instance with SSH and then install docker. Follow the steps below to set up the environment easily. For reproducibility, we list the versions and commits we are using in the commands.

\n
    \n
  1. $ git clone https://github.com/huggingface/transformers.git
    2. \n
  2. $ cd transformers/
    3. \n
  3. $ git checkout 26b65fb5168f324277b85c558ef8209bfceae1fe
    4. \n
  4. $ cd docker/transformers-intel-cpu/
    5. \n
  5. $ sudo docker build . -t <your_docker_image_tag>
    6. \n
  6. $ sudo docker run -it --rm --privileged -v /home/<your_home_folder>:/workspace <your_docker_image_tag> /bin/bash
    7. \n
\n

We are in container now, do following steps.

\n
    \n
  1. $ pip install git+https://github.com/huggingface/transformers.git@26b65fb5168f324277b85c558ef8209bfceae1fe
    2. \n
  2. $ pip install torch==2.8.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
    3. \n
\n

Benchmark Procedure

\n

For each batch size we

\n
    \n
  1. Build a fixed-length 1024‑token left‑padded batch.
    2. \n
  2. Run a single warm‑up round.
    3. \n
  3. set max_new_tokens=1024 and measure total latency, then get throughput=(OUTPUT_TOKENS∗batch_size)/total_latencythroughput = (OUTPUT\\_TOKENS * batch\\_size) / total\\_latency.
    4. \n
\n

Run numactl -l python benchmark.py for the following codes.

\n
import os\nimport time\nimport torch\nfrom datasets import load_dataset\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nINPUT_TOKENS = 1024\nOUTPUT_TOKENS = 1024\n\ndef get_inputs(tokenizer, batch_size):\n    dataset = load_dataset("ola13/small-the_pile", split="train")\n    tokenizer.padding_side = "left"\n    selected_texts = []\n    for sample in dataset:\n        input_ids = tokenizer(sample["text"], return_tensors="pt").input_ids\n        if len(selected_texts) == 0 and input_ids.shape[-1] >= INPUT_TOKENS:\n            selected_texts.append(sample["text"])\n        elif len(selected_texts) > 0:\n            selected_texts.append(sample["text"])\n        if len(selected_texts) == batch_size:\n            break\n\n    return tokenizer(selected_texts, max_length=INPUT_TOKENS, padding="max_length", truncation=True, return_tensors="pt")\n\ndef run_generate(model, inputs, generation_config):\n    inputs["generation_config"] = generation_config\n    model.generate(**inputs) # warm up\n    pre = time.time()\n    model.generate(**inputs)\n    latency = (time.time() - pre)\n    return latency\n\ndef benchmark(model, tokenizer, batch_size, generation_config):\n    inputs = get_inputs(tokenizer, batch_size)\n    generation_config.max_new_tokens = 1\n    generation_config.min_new_tokens = 1\n    prefill_latency = run_generate(model, inputs, generation_config)\n    generation_config.max_new_tokens = OUTPUT_TOKENS\n    generation_config.min_new_tokens = OUTPUT_TOKENS\n    total_latency = run_generate(model, inputs, generation_config)\n    decoding_latency = (total_latency - prefill_latency) / (OUTPUT_TOKENS - 1)\n    throughput = OUTPUT_TOKENS * batch_size / total_latency\n\n    return prefill_latency, decoding_latency, throughput\n\n\nif __name__ == "__main__":\n    model_id = "unsloth/gpt-oss-120b-BF16"\n    tokenizer = AutoTokenizer.from_pretrained(model_id)\n    model_kwargs = {"dtype": torch.bfloat16}\n    model = AutoModelForCausalLM.from_pretrained(model_id, **model_kwargs)\n    model.config._attn_implementation="sdpa"\n    generation_config = model.generation_config\n    generation_config.do_sample = False\n    generation_config.cache_implementation="static"\n\n    for batch_size in [1, 2, 4, 8, 16, 32, 64]:\n        print(f"---------- Run generation with batch size = {batch_size} ----------", flush=True)\n        prefill_latency, decoding_latency, throughput = benchmark(model, tokenizer, batch_size, generation_config)\n        print(f"throughput = {throughput}", flush=True)\n
\n

Results

\n

Normalized Throughput per vCPU

\n

Across batch sizes up to 64, Intel Xeon 6 processor‑powered C4 consistently outperforms C3 with a 1.4x to 1.7× throughput per-vCPU. The formula is:

\n

normalized_throughput_per_vCPU=throughput_C4/vCPUs_C4throughput_C3/vCPUs_C3 \\text{normalized\\_throughput\\_per\\_vCPU} = \\frac{\\text{throughput\\_C4} / \\text{vCPUs\\_C4}} {\\text{throughput\\_C3} / \\text{vCPUs\\_C3}}

\n

\"throughput-gpt-oss-per-vcpu\"

\n

Cost & TCO

\n

At batch size 64, C4 provides 1.7× the per‑vCPU throughput of C3; with near parity in price per vCPU (hourly cost scales linearly with vCPU count), this yields a 1.7× TCO advantage (C3 would require 1.7× the spend for the same generated token volume).

\n

Per‑vCPU throughput ratio: throughput_C4/vCPUs_C4throughput_C3/vCPUs_C3=1.7⇒TCO_C3TCO_C4≈1.7 \\frac{\\text{throughput\\_C4} / \\text{vCPUs\\_C4}}{\\text{throughput\\_C3} / \\text{vCPUs\\_C3}} = 1.7 \\Rightarrow \\frac{\\text{TCO\\_C3}}{\\text{TCO\\_C4}} \\approx 1.7

\n

\"throughput-gpt-oss-per-dollar\"

\n

Conclusion

\n

Google Cloud C4 VMs powered by Intel Xeon 6 processors (GNR) provide both impressive performance gains and better cost efficiency for large MoE inference over previous generation Google Cloud C3 VM (powered by 4th Gen Intel Xeon processors). For GPT OSS MoE inference, we observed combined higher throughput, lower latency, and reduced cost. These results underline that thanks to targeted framework optimizations from Intel and Hugging Face, large MoE models can be efficiently served on next-generation general-purpose CPUs.

\n", "content_markdown": "Intel and Hugging Face collaborated to demonstrate the real-world value of upgrading to Google’s latest `C4` Virtual Machine (VM) running on Intel® Xeon® 6 processors (codenamed Granite Rapids (GNR)). We specifically wanted to benchmark improvements in the text generation performance of OpenAI GPT OSS Large Language Model(LLM).\n\nThe results are in, and they are impressive, demonstrating a *1.7x* improvement in Total Cost of Ownership(TCO) over the previous-generation Google `C3` VM instances. The Google Cloud `C4` VM instance further resulted in:\n\n- 1.4x to 1.7x TPOT throughput/vCPU/dollar\n- Lower price per hour over `C3` VM\n\n## Introduction\n\nGPT OSS is a common name for an open-source Mixture of Experts (MoE) model released by OpenAI. An MoE model is a deep neural network architecture that uses specialized “expert” sub-networks and a “gating network” to decide which experts to use for a given input. MoE models allow you to scale your model capacity efficiently without linearly scaling compute costs. They also allow for specialization, where different “experts” learn different skills, allowing them to adapt to diverse data distributions.\n\nEven with very large parameters, only a small subset of experts is activated per token, making CPU inference viable.\n\nIntel and Hugging Face collaborated to merge an expert execution optimization (PR [#40304](https://github.com/huggingface/transformers/pull/40304)) to eliminate redundant computation where every expert processed all tokens to transformers. This optimization directed each expert to run only on the tokens it is routed to, removing FLOPs waste and improving utilization.\n\n![gpt_oss_expert](https://huggingface.co/datasets/Intel/blog/resolve/main/gpt-oss-on-intel-xeon/gpt_oss_expert.png)\n\n## Benchmark Scope & Hardware\n\nWe benchmarked GPT OSS under a controlled, repeatable generation workload to isolate architectural differences (GCP `C4` VMs on Intel Xeon 6 processors (GNR) vs GCP `C3` VMs on 4th Gen Intel Xeon Processors (SPR)) and MoE execution efficiency. The focus is steady‑state decoding (per‑token latency) and end‑to‑end normalized throughput with increasing batch size while keeping sequence lengths fixed. All runs use static KV cache and SDPA attention for determinism.\n\n### Configuration Summary\n\n- Model: [unsloth/gpt-oss-120b-BF16](https://huggingface.co/unsloth/gpt-oss-120b-BF16)\n- Precision: bfloat16\n- Task: Text generation\n- Input length: 1024 tokens (left‑padded)\n- Output length: 1024 tokens\n- Batch sizes: 1, 2, 4, 8, 16, 32, 64\n- Enabled features:\n - Static KV cache\n - SDPA attention backend\n- Reported metrics:\n - Throughput (Total generated tokens per second aggregated over the batch)\n\n### Hardware Under Test\n\n| Instance | Architecture | vCPUs |\n| -------- | ---------------------------------- | ----- |\n| `C3` | 4th Gen Intel Xeon processor (SPR) | 172 |\n| `C4` | Intel Xeon 6 processor (GNR) | 144 |\n\n## Create instance\n\n### `C3`\n\nVisit [`Google Cloud Console`](https://console.cloud.google.com/) and click on `create a VM` under your project. Follow the steps below to create a `176 vCPU` instance.\n\n1. pick `C3` in the `Machine configuration` and specify Machine type as `c3-standard-176`. You also need to set the `CPU platform` and turn on `all-core turbo` to make performance more stable: [![alt text](https://huggingface.co/datasets/Intel/blog/resolve/main/gpt-oss-on-intel-xeon/spr.png)](https://huggingface.co/datasets/Intel/blog/resolve/main/gpt-oss-on-intel-xeon/spr.png)\n2. configure OS and storage tab as below: [![alt text](https://huggingface.co/datasets/Intel/blog/resolve/main/gpt-oss-on-intel-xeon/spr-os.png)](https://huggingface.co/datasets/Intel/blog/resolve/main/gpt-oss-on-intel-xeon/spr-os.png)\n3. keep other configurations as default\n4. click `Create` button\n\n### `C4`\n\nVisit [`Google Cloud Console`](https://console.cloud.google.com/) and click on `create a VM` under your project. Follow the below steps to create a `144 vCPU` instance.\n\n1. pick `C4` in the `Machine configuration` tab and specify Machine type as `c4-standard-144`. You can also set the `CPU platform` and turn on all-core turbo to make performance more stable: [![alt text](https://huggingface.co/datasets/Intel/blog/resolve/main/gpt-oss-on-intel-xeon/gnr.png)](https://huggingface.co/datasets/Intel/blog/resolve/main/gpt-oss-on-intel-xeon/gnr.png)\n2. configure OS and storage tab as we need for C3.\n3. keep other configurations as default\n4. click `Create` button\n\n## Set up the environment\n\nLogin the instance with SSH and then install docker. Follow the steps below to set up the environment easily. For reproducibility, we list the versions and commits we are using in the commands.\n\n1. `$ git clone https://github.com/huggingface/transformers.git`\n2. `$ cd transformers/`\n3. `$ git checkout 26b65fb5168f324277b85c558ef8209bfceae1fe`\n4. `$ cd docker/transformers-intel-cpu/`\n5. `$ sudo docker build . -t `\n6. `$ sudo docker run -it --rm --privileged -v /home/:/workspace /bin/bash`\n\nWe are in container now, do following steps.\n\n1. `$ pip install git+https://github.com/huggingface/transformers.git@26b65fb5168f324277b85c558ef8209bfceae1fe`\n2. `$ pip install torch==2.8.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu`\n\n## Benchmark Procedure\n\nFor each batch size we\n\n1. Build a fixed-length 1024‑token left‑padded batch.\n2. Run a single warm‑up round.\n3. set `max_new_tokens=1024` and measure total latency, then get throughput=(OUTPUT\\_TOKENS∗batch\\_size)/total\\_latencythroughput = (OUTPUT\\\\\\_TOKENS \\* batch\\\\\\_size) / total\\\\\\_latency.\n\nRun `numactl -l python benchmark.py` for the following codes.\n\n```python\nimport os\nimport time\nimport torch\nfrom datasets import load_dataset\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nINPUT_TOKENS = 1024\nOUTPUT_TOKENS = 1024\n\ndef get_inputs(tokenizer, batch_size):\n dataset = load_dataset(\"ola13/small-the_pile\", split=\"train\")\n tokenizer.padding_side = \"left\"\n selected_texts = []\n for sample in dataset:\n input_ids = tokenizer(sample[\"text\"], return_tensors=\"pt\").input_ids\n if len(selected_texts) == 0 and input_ids.shape[-1] >= INPUT_TOKENS:\n selected_texts.append(sample[\"text\"])\n elif len(selected_texts) > 0:\n selected_texts.append(sample[\"text\"])\n if len(selected_texts) == batch_size:\n break\n\n return tokenizer(selected_texts, max_length=INPUT_TOKENS, padding=\"max_length\", truncation=True, return_tensors=\"pt\")\n\ndef run_generate(model, inputs, generation_config):\n inputs[\"generation_config\"] = generation_config\n model.generate(**inputs) # warm up\n pre = time.time()\n model.generate(**inputs)\n latency = (time.time() - pre)\n return latency\n\ndef benchmark(model, tokenizer, batch_size, generation_config):\n inputs = get_inputs(tokenizer, batch_size)\n generation_config.max_new_tokens = 1\n generation_config.min_new_tokens = 1\n prefill_latency = run_generate(model, inputs, generation_config)\n generation_config.max_new_tokens = OUTPUT_TOKENS\n generation_config.min_new_tokens = OUTPUT_TOKENS\n total_latency = run_generate(model, inputs, generation_config)\n decoding_latency = (total_latency - prefill_latency) / (OUTPUT_TOKENS - 1)\n throughput = OUTPUT_TOKENS * batch_size / total_latency\n\n return prefill_latency, decoding_latency, throughput\n\n\nif __name__ == \"__main__\":\n model_id = \"unsloth/gpt-oss-120b-BF16\"\n tokenizer = AutoTokenizer.from_pretrained(model_id)\n model_kwargs = {\"dtype\": torch.bfloat16}\n model = AutoModelForCausalLM.from_pretrained(model_id, **model_kwargs)\n model.config._attn_implementation=\"sdpa\"\n generation_config = model.generation_config\n generation_config.do_sample = False\n generation_config.cache_implementation=\"static\"\n\n for batch_size in [1, 2, 4, 8, 16, 32, 64]:\n print(f\"---------- Run generation with batch size = {batch_size} ----------\", flush=True)\n prefill_latency, decoding_latency, throughput = benchmark(model, tokenizer, batch_size, generation_config)\n print(f\"throughput = {throughput}\", flush=True)\n```\n\n## Results\n\n### Normalized Throughput per vCPU\n\nAcross batch sizes up to 64, Intel Xeon 6 processor‑powered `C4` consistently outperforms `C3` with a 1.4x to 1.7× throughput per-vCPU. The formula is:\n\nnormalized\\_throughput\\_per\\_vCPU=throughput\\_C4/vCPUs\\_C4throughput\\_C3/vCPUs\\_C3 \\\\text{normalized\\\\\\_throughput\\\\\\_per\\\\\\_vCPU} = \\\\frac{\\\\text{throughput\\\\\\_C4} / \\\\text{vCPUs\\\\\\_C4}} {\\\\text{throughput\\\\\\_C3} / \\\\text{vCPUs\\\\\\_C3}}\n\n![throughput-gpt-oss-per-vcpu](https://huggingface.co/datasets/Intel/blog/resolve/main/gpt-oss-on-intel-xeon/throughput-gpt-oss-per-vcpu.png)\n\n### Cost & TCO\n\nAt batch size 64, `C4` provides 1.7× the per‑vCPU throughput of `C3`; with near parity in price per vCPU (hourly cost scales linearly with vCPU count), this yields a 1.7× TCO advantage (`C3` would require 1.7× the spend for the same generated token volume).\n\nPer‑vCPU throughput ratio: throughput\\_C4/vCPUs\\_C4throughput\\_C3/vCPUs\\_C3=1.7⇒TCO\\_C3TCO\\_C4≈1.7 \\\\frac{\\\\text{throughput\\\\\\_C4} / \\\\text{vCPUs\\\\\\_C4}}{\\\\text{throughput\\\\\\_C3} / \\\\text{vCPUs\\\\\\_C3}} = 1.7 \\\\Rightarrow \\\\frac{\\\\text{TCO\\\\\\_C3}}{\\\\text{TCO\\\\\\_C4}} \\\\approx 1.7\n\n![throughput-gpt-oss-per-dollar](https://huggingface.co/datasets/Intel/blog/resolve/main/gpt-oss-on-intel-xeon/throughput-gpt-oss-per-dollar.png)\n\n## Conclusion\n\nGoogle Cloud `C4` VMs powered by Intel Xeon 6 processors (GNR) provide both impressive performance gains and better cost efficiency for large MoE inference over previous generation Google Cloud `C3` VM (powered by 4th Gen Intel Xeon processors). For GPT OSS MoE inference, we observed combined higher throughput, lower latency, and reduced cost. These results underline that thanks to targeted framework optimizations from Intel and Hugging Face, large MoE models can be efficiently served on next-generation general-purpose CPUs.\n", "dateCreated": "2026-10-04T12:50:30.804479567+00:00", "date_created": "2025-10-16T00:00:00+00:00", "date_modified": "2025-10-16T00:00:00+00:00", "date_published": "2025-10-16T00:00:00+00:00", "description": "Intel and Hugging Face collaborated to demonstrate the real-world value of upgrading to Google’s latest C4 Virtual Machine (VM) running on Intel® Xeon® 6 processors (codenamed Granite Rapids (GNR)). We specifically wanted to benchmark…", "external_url": "https://huggingface.co/blog/gpt-oss-on-intel-xeon", "git": { "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e", "path": "items/huggingface-co/2025/10/2025-10-16-google-cloud-c4-brings-a-70-tco-improvement-on-gpt-oss-with.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/huggingface-co/2025/10/2025-10-16-google-cloud-c4-brings-a-70-tco-improvement-on-gpt-oss-with.md" }, "id": "urn:aggr:item:4721c25a9af84806b28e2392e9e304d1d50fc0cc", "isBasedOn": { "@id": "https://huggingface.co/blog/gpt-oss-on-intel-xeon" }, "mainEntity": { "@id": "https://huggingface.co/blog/gpt-oss-on-intel-xeon", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2025-10-16-google-cloud-c4-brings-a-70-tco-improvement-on-gpt-oss-with/#webpage" }, "dateModified": "2025-10-16T00:00:00+00:00", "datePublished": "2025-10-16T00:00:00+00:00", "headline": "Google Cloud C4 Brings a 70% TCO improvement on GPT OSS with Intel and Hugging Face", "timeRequired": "PT5M", "url": "https://huggingface.co/blog/gpt-oss-on-intel-xeon", "wordCount": 1109 }, "name": "Archived snapshot: Google Cloud C4 Brings a 70% TCO improvement on GPT OSS with Intel and Hugging Face", "preview": null, "reading_minutes": 5, "replicated_at": null, "source": "huggingface-co", "source_name": "Hugging Face - Blog", "summary": null, "tags": [], "temporalCoverage": "2026-10-04T12:50:30.804479567+00:00", "title": "Google Cloud C4 Brings a 70% TCO improvement on GPT OSS with Intel and Hugging Face", "url": "https://aggr.aymericbeaumet.com/items/huggingface-co/2025-10-16-google-cloud-c4-brings-a-70-tco-improvement-on-gpt-oss-with/", "word_count": 1109 }