{ "@context": "https://schema.org", "@id": "https://aggr.aymericbeaumet.com/items/philschmid-de/2024-12-03-deploy-qwq-32b-preview-the-best-open-reasoning-model-on-aws/#webpage", "@type": [ "WebPage", "ArchiveComponent" ], "articleBody": "[QwQ-32B-Preview](https://huggingface.co/Qwen/QwQ-32B-Preview), developed by the Qwen team at Alibaba and available on Hugging Face, is the best open reasoning model for mathematical and programming reasoning capabilities among open models directly competing with OpenAI o1.\n\n![QwQ](https://www.philschmid.de/static/blog/sagemaker-deploy-qwq/qwq.png)\n\n- **Mathematical Reasoning**: Achieves an impressive 90.6% on MATH-500, outperforming both Claude 3.5 (78.3%) and matching OpenAI's o1-mini (90.0%)\n- **Advanced Mathematics**: Scores 50.0% on AIME (American Invitational Mathematics Examination), significantly 'higher than Claude 3.5 (16.0%)\n- **Scientific Reasoning**: Demonstrates strong performance on GPQA with 65.2%, on par with Claude 3.5 (65.0%)\n- **Programming**: Achieves 50.0% on LiveCodeBench, showing competitive performance with leading proprietary models\n\nIn this guide, you'll learn how to deploy QwQ model on Amazon SageMaker using the Hugging Face LLM DLC (Deep Learning Container). The DLC is powered by [Text Generation Inference (TGI)](https://github.com/huggingface/text-generation-inference), providing an optimized, production-ready environment for serving Large Language Models.\n\n> \\[!NOTE\\] QwQ-32B-Preview is released under the Apache 2.0 license, making it suitable for both research and commercial applications.\n\nWe'll cover:\n\n1. [Setup development environment](https://www.philschmid.de/sagemaker-deploy-qwq#1-setup-development-environment)\n2. [Retrieve the new Hugging Face LLM DLC](https://www.philschmid.de/sagemaker-deploy-qwq#2-retrieve-the-new-hugging-face-llm-dlc)\n3. [Deploy QwQ-32B-Preview to Amazon SageMaker](https://www.philschmid.de/sagemaker-deploy-qwq#3-deploy-qwq-32b-preview-to-amazon-sagemaker)\n4. [Run reasoning with QwQ and solve complex math problems](https://www.philschmid.de/sagemaker-deploy-qwq#4-run-reasoning-with-qwq-and-solve-complex-math-problems)\n\nLet's get started deploying one of the most capable open-source reasoning models available today!\n\n## 1\\. Setup development environment\n\nWe are going to use the `sagemaker` python SDK to deploy QwQ to Amazon SageMaker. We need to make sure to have an AWS account configured and the `sagemaker` python SDK installed.\n\nPython\n\n```python\n!pip install \"sagemaker>=2.232.2\" --upgrade --quiet\n \n```\n\nIf you are going to use Sagemaker in a local environment. You need access to an IAM Role with the required permissions for Sagemaker. You can find [here](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-roles.html) more about it.\n\nPython\n\n```python\nimport sagemaker\nimport boto3\nsess = sagemaker.Session()\n# sagemaker session bucket -> used for uploading data, models and logs\n# sagemaker will automatically create this bucket if it not exists\nsagemaker_session_bucket=None\nif sagemaker_session_bucket is None and sess is not None:\n # set to default bucket if a bucket name is not given\n sagemaker_session_bucket = sess.default_bucket()\n \ntry:\n role = sagemaker.get_execution_role()\nexcept ValueError:\n iam = boto3.client('iam')\n role = iam.get_role(RoleName='sagemaker_execution_role')['Role']['Arn']\n \nsess = sagemaker.Session(default_bucket=sagemaker_session_bucket)\n \nprint(f\"sagemaker role arn: {role}\")\nprint(f\"sagemaker session region: {sess.boto_region_name}\")\n \n```\n\n## 2\\. Retrieve the new Hugging Face LLM DLC\n\nCompared to deploying regular Hugging Face models we first need to retrieve the container uri and provide it to our `HuggingFaceModel` model class with a `image_uri` pointing to the image. To retrieve the new Hugging Face LLM DLC in Amazon SageMaker, we can use the `get_huggingface_llm_image_uri` method provided by the `sagemaker` SDK. This method allows us to retrieve the URI for the desired Hugging Face LLM DLC based on the specified `backend`, `session`, `region`, and `version`. You can find the available versions [here](https://github.com/aws/deep-learning-containers/blob/master/available_images.md#huggingface-text-generation-inference-containers)\n\nPython\n\n```python\n# TODO: COMMENT IN WHEN IMAGE IS RELEASED\n# from sagemaker.huggingface import get_huggingface_llm_image_uri\n \n# # retrieve the llm image uri\n# llm_image = get_huggingface_llm_image_uri(\n# \"huggingface\",\n# version=\"2.4.0\"\n# )\n \n# # print ecr image uri\n# print(f\"llm image uri: {llm_image}\")\n \nllm_image = f\"763104351884.dkr.ecr.{sess.boto_region_name}.amazonaws.com/huggingface-pytorch-tgi-inference:2.4-tgi2.3-gpu-py311-cu124-ubuntu22.04\"\n \n```\n\n## 3\\. Deploy QwQ-32B-Preview to Amazon SageMaker\n\nTo deploy [Qwen/QwQ-32B-Preview](https://huggingface.co/Qwen/QwQ-32B-Preview) to Amazon SageMaker we create a `HuggingFaceModel` model class and define our endpoint configuration including the `hf_model_id`, `instance_type` etc. We will use a `g6.12xlarge` instance type, which has 4 NVIDIA L4 GPUs and 96GB of GPU memory.\n\nQwQ-32B is a 32B parameter big dense decoder requiring ~64GB of raw GPU memory to load it + additional PyTorch overhead and storage for the KV-Cache Storage.\n\n| Model | Instance Type | \\# of GPUs per replica | quantization |\n| ------------------------------------------------------------------ | --------------------- | ---------------------- | ------------ |\n| [QwQ-32B-Preview](https://www.philschmid.de/Qwen/QwQ-32B-Preview) | `(ml.)g6e.2xlarge` | 1 | int4   |\n| [QwQ-32B-Preview](https://www.philschmid.de/Qwen/QwQ-32B-Preview) | `(ml.)g6e.2xlarge` | 1 | fp8   |\n| [QwQ-32B-Preview](https://www.philschmid.de/Qwen/QwQ-32B-Preview) | `(ml.)g5/g6.12xlarge` | 4 | \\-   |\n| [QwQ-32B-Preview](https://www.philschmid.de/Qwen/QwQ-32B-Preview) | `(ml.)g6e.12xlarge` | 4 | \\-   |\n| [QwQ-32B-Preview](https://www.philschmid.de/Qwen/QwQ-32B-Preview) | `(ml.)p4d.24xlarge` | 8 | \\-   |\n\nWe are going to use the `g6.12xlarge` instance type with 4 GPUs.\n\nPython\n\n```python\nimport json\nfrom sagemaker.huggingface import HuggingFaceModel\n \n# sagemaker config\ninstance_type = \"ml.g6.12xlarge\"\nnumber_of_gpu = 4\nhealth_check_timeout = 300\n \n# Define Model and Endpoint configuration parameter\nconfig = {\n 'HF_MODEL_ID': \"Qwen/QwQ-32B-Preview\", # model_id from hf.co/models\n 'SM_NUM_GPUS': json.dumps(number_of_gpu), # Number of GPU used per replica\n 'MAX_INPUT_LENGTH': json.dumps(4096), # Max length of input text\n 'MAX_TOTAL_TOKENS': json.dumps(8192), # Max length of the generation (including input text)\n 'HF_HUB_ENABLE_HF_TRANSFER': \"1\", # Enable HF transfer for faster downloads\n 'MESSAGES_API_ENABLED': \"true\", # Enable OpenAI compatible messages API\n}\n \n# create HuggingFaceModel with the image uri\nllm_model = HuggingFaceModel(\n role=role,\n image_uri=llm_image,\n env=config\n)\n```\n\nAfter we have created the `HuggingFaceModel` we can deploy it to Amazon SageMaker using the `deploy` method. We will deploy the model with the `ml.g6.12xlarge` instance type. TGI will automatically distribute and shard the model across all GPUs.\n\nPython\n\n```python\n# Deploy model to an endpoint\nllm = llm_model.deploy(\n initial_instance_count=1,\n instance_type=instance_type,\n container_startup_health_check_timeout=health_check_timeout, # 10 minutes to be able to load the model\n)\n \n```\n\nSageMaker will now create our endpoint and deploy the model to it. This can takes a 10-15 minutes.\n\n## 4\\. Run reasoning with QwQ and solve complex math problems\n\nAfter our endpoint is deployed we can run inference on it. We will use the `predict` method from the `predictor` to run inference on our endpoint. We create a helper [stream\\_request.py](https://www.philschmid.de/demo/stream_request.py) to stream tokens. This makes it easier to follow the reasoning process.\n\nQwQ is trained for advancing AI reasoning problems, like complex math problems using chain of thought similar to OpenAI's o1. We added small helper util to stream the response as it generates a lot of tokens to solve the problems.\n\nPython\n\n```python\nimport sys\nsys.path.append(\"../demo\") \nfrom stream_request import create_streamer\n \nstreamer = create_streamer(llm.endpoint_name)\n \nprompt = \"How many r in strawberry.\"\nres = streamer(prompt, max_tokens=4096)\n \nfor chunk in res:\n print(chunk, end=\"\", flush=True)\n```\n\n``\n\n`Let's see. The word is \"strawberry.\" I need to find out how many 'r's are in it. Okay, first, I'll spell it out slowly: s-t-r-a-w-b-e-r-r-y. Okay, now, I'll count the 'r's. Let's see: there's an 'r' after the 't', then another 'r' towards the end, and then another one after that. Wait, let's check again. s-t-r-a-w-b-e-r-r-y. So, the first 'r' is the third letter, then there's another 'r' before the last letter, and another one right after it. So, that's three 'r's in \"strawberry.\" But, maybe I should double-check because it's easy to miss letters when counting. Let me write it down: s-t-r-a-w-b-e-r-r-y. Now, I'll point to each 'r' one by one. First 'r' here, second 'r' here, and third 'r' here. Yep, three 'r's in \"strawberry.\" I think that's correct. [Final Answer] 3`\n\n## 5\\. Clean up\n\nTo clean up, we can delete the model and endpoint.\n", "authors": [], "captured_at": "2026-09-27T19:29:15.293681927+00:00", "category": "programming", "content_html": "

QwQ-32B-Preview, developed by the Qwen team at Alibaba and available on Hugging Face, is the best open reasoning model for mathematical and programming reasoning capabilities among open models directly competing with OpenAI o1.

\n

\"QwQ\"

\n\n

In this guide, you'll learn how to deploy QwQ model on Amazon SageMaker using the Hugging Face LLM DLC (Deep Learning Container). The DLC is powered by Text Generation Inference (TGI), providing an optimized, production-ready environment for serving Large Language Models.

\n
\n

[!NOTE] QwQ-32B-Preview is released under the Apache 2.0 license, making it suitable for both research and commercial applications.

\n
\n

We'll cover:

\n
    \n
  1. Setup development environment
    2. \n
  2. Retrieve the new Hugging Face LLM DLC
    3. \n
  3. Deploy QwQ-32B-Preview to Amazon SageMaker
    4. \n
  4. Run reasoning with QwQ and solve complex math problems
    5. \n
\n

Let's get started deploying one of the most capable open-source reasoning models available today!

\n

1. Setup development environment

\n

We are going to use the sagemaker python SDK to deploy QwQ to Amazon SageMaker. We need to make sure to have an AWS account configured and the sagemaker python SDK installed.

\n

Python

\n
!pip install "sagemaker>=2.232.2" --upgrade --quiet\n \n
\n

If you are going to use Sagemaker in a local environment. You need access to an IAM Role with the required permissions for Sagemaker. You can find here more about it.

\n

Python

\n
import sagemaker\nimport boto3\nsess = sagemaker.Session()\n# sagemaker session bucket -> used for uploading data, models and logs\n# sagemaker will automatically create this bucket if it not exists\nsagemaker_session_bucket=None\nif sagemaker_session_bucket is None and sess is not None:\n    # set to default bucket if a bucket name is not given\n    sagemaker_session_bucket = sess.default_bucket()\n \ntry:\n    role = sagemaker.get_execution_role()\nexcept ValueError:\n    iam = boto3.client('iam')\n    role = iam.get_role(RoleName='sagemaker_execution_role')['Role']['Arn']\n \nsess = sagemaker.Session(default_bucket=sagemaker_session_bucket)\n \nprint(f"sagemaker role arn: {role}")\nprint(f"sagemaker session region: {sess.boto_region_name}")\n \n
\n

2. Retrieve the new Hugging Face LLM DLC

\n

Compared to deploying regular Hugging Face models we first need to retrieve the container uri and provide it to our HuggingFaceModel model class with a image_uri pointing to the image. To retrieve the new Hugging Face LLM DLC in Amazon SageMaker, we can use the get_huggingface_llm_image_uri method provided by the sagemaker SDK. This method allows us to retrieve the URI for the desired Hugging Face LLM DLC based on the specified backend, session, region, and version. You can find the available versions here

\n

Python

\n
# TODO: COMMENT IN WHEN IMAGE IS RELEASED\n# from sagemaker.huggingface import get_huggingface_llm_image_uri\n \n# # retrieve the llm image uri\n# llm_image = get_huggingface_llm_image_uri(\n#   "huggingface",\n#   version="2.4.0"\n# )\n \n# # print ecr image uri\n# print(f"llm image uri: {llm_image}")\n \nllm_image = f"763104351884.dkr.ecr.{sess.boto_region_name}.amazonaws.com/huggingface-pytorch-tgi-inference:2.4-tgi2.3-gpu-py311-cu124-ubuntu22.04"\n \n
\n

3. Deploy QwQ-32B-Preview to Amazon SageMaker

\n

To deploy Qwen/QwQ-32B-Preview to Amazon SageMaker we create a HuggingFaceModel model class and define our endpoint configuration including the hf_model_id, instance_type etc. We will use a g6.12xlarge instance type, which has 4 NVIDIA L4 GPUs and 96GB of GPU memory.

\n

QwQ-32B is a 32B parameter big dense decoder requiring ~64GB of raw GPU memory to load it + additional PyTorch overhead and storage for the KV-Cache Storage.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ModelInstance Type# of GPUs per replicaquantization
QwQ-32B-Preview(ml.)g6e.2xlarge1int4  
QwQ-32B-Preview(ml.)g6e.2xlarge1fp8  
QwQ-32B-Preview(ml.)g5/g6.12xlarge4-  
QwQ-32B-Preview(ml.)g6e.12xlarge4-  
QwQ-32B-Preview(ml.)p4d.24xlarge8-  
\n

We are going to use the g6.12xlarge instance type with 4 GPUs.

\n

Python

\n
import json\nfrom sagemaker.huggingface import HuggingFaceModel\n \n# sagemaker config\ninstance_type = "ml.g6.12xlarge"\nnumber_of_gpu = 4\nhealth_check_timeout = 300\n \n# Define Model and Endpoint configuration parameter\nconfig = {\n  'HF_MODEL_ID': "Qwen/QwQ-32B-Preview", # model_id from hf.co/models\n  'SM_NUM_GPUS': json.dumps(number_of_gpu), # Number of GPU used per replica\n  'MAX_INPUT_LENGTH': json.dumps(4096),  # Max length of input text\n  'MAX_TOTAL_TOKENS': json.dumps(8192),  # Max length of the generation (including input text)\n  'HF_HUB_ENABLE_HF_TRANSFER': "1", # Enable HF transfer for faster downloads\n  'MESSAGES_API_ENABLED': "true", # Enable OpenAI compatible messages API\n}\n \n# create HuggingFaceModel with the image uri\nllm_model = HuggingFaceModel(\n  role=role,\n  image_uri=llm_image,\n  env=config\n)\n
\n

After we have created the HuggingFaceModel we can deploy it to Amazon SageMaker using the deploy method. We will deploy the model with the ml.g6.12xlarge instance type. TGI will automatically distribute and shard the model across all GPUs.

\n

Python

\n
# Deploy model to an endpoint\nllm = llm_model.deploy(\n  initial_instance_count=1,\n  instance_type=instance_type,\n  container_startup_health_check_timeout=health_check_timeout, # 10 minutes to be able to load the model\n)\n \n
\n

SageMaker will now create our endpoint and deploy the model to it. This can takes a 10-15 minutes.

\n

4. Run reasoning with QwQ and solve complex math problems

\n

After our endpoint is deployed we can run inference on it. We will use the predict method from the predictor to run inference on our endpoint. We create a helper stream_request.py to stream tokens. This makes it easier to follow the reasoning process.

\n

QwQ is trained for advancing AI reasoning problems, like complex math problems using chain of thought similar to OpenAI's o1. We added small helper util to stream the response as it generates a lot of tokens to solve the problems.

\n

Python

\n
import sys\nsys.path.append("../demo") \nfrom stream_request import create_streamer\n \nstreamer = create_streamer(llm.endpoint_name)\n \nprompt = "How many r in strawberry."\nres = streamer(prompt, max_tokens=4096)\n \nfor chunk in res:\n    print(chunk, end="", flush=True)\n
\n

``

\n

Let's see. The word is "strawberry." I need to find out how many 'r's are in it. Okay, first, I'll spell it out slowly: s-t-r-a-w-b-e-r-r-y. Okay, now, I'll count the 'r's. Let's see: there's an 'r' after the 't', then another 'r' towards the end, and then another one after that. Wait, let's check again. s-t-r-a-w-b-e-r-r-y. So, the first 'r' is the third letter, then there's another 'r' before the last letter, and another one right after it. So, that's three 'r's in "strawberry." But, maybe I should double-check because it's easy to miss letters when counting. Let me write it down: s-t-r-a-w-b-e-r-r-y. Now, I'll point to each 'r' one by one. First 'r' here, second 'r' here, and third 'r' here. Yep, three 'r's in "strawberry." I think that's correct. [Final Answer] 3

\n

5. Clean up

\n

To clean up, we can delete the model and endpoint.

\n", "content_markdown": "[QwQ-32B-Preview](https://huggingface.co/Qwen/QwQ-32B-Preview), developed by the Qwen team at Alibaba and available on Hugging Face, is the best open reasoning model for mathematical and programming reasoning capabilities among open models directly competing with OpenAI o1.\n\n![QwQ](https://www.philschmid.de/static/blog/sagemaker-deploy-qwq/qwq.png)\n\n- **Mathematical Reasoning**: Achieves an impressive 90.6% on MATH-500, outperforming both Claude 3.5 (78.3%) and matching OpenAI's o1-mini (90.0%)\n- **Advanced Mathematics**: Scores 50.0% on AIME (American Invitational Mathematics Examination), significantly 'higher than Claude 3.5 (16.0%)\n- **Scientific Reasoning**: Demonstrates strong performance on GPQA with 65.2%, on par with Claude 3.5 (65.0%)\n- **Programming**: Achieves 50.0% on LiveCodeBench, showing competitive performance with leading proprietary models\n\nIn this guide, you'll learn how to deploy QwQ model on Amazon SageMaker using the Hugging Face LLM DLC (Deep Learning Container). The DLC is powered by [Text Generation Inference (TGI)](https://github.com/huggingface/text-generation-inference), providing an optimized, production-ready environment for serving Large Language Models.\n\n> \\[!NOTE\\] QwQ-32B-Preview is released under the Apache 2.0 license, making it suitable for both research and commercial applications.\n\nWe'll cover:\n\n1. [Setup development environment](https://www.philschmid.de/sagemaker-deploy-qwq#1-setup-development-environment)\n2. [Retrieve the new Hugging Face LLM DLC](https://www.philschmid.de/sagemaker-deploy-qwq#2-retrieve-the-new-hugging-face-llm-dlc)\n3. [Deploy QwQ-32B-Preview to Amazon SageMaker](https://www.philschmid.de/sagemaker-deploy-qwq#3-deploy-qwq-32b-preview-to-amazon-sagemaker)\n4. [Run reasoning with QwQ and solve complex math problems](https://www.philschmid.de/sagemaker-deploy-qwq#4-run-reasoning-with-qwq-and-solve-complex-math-problems)\n\nLet's get started deploying one of the most capable open-source reasoning models available today!\n\n## 1\\. Setup development environment\n\nWe are going to use the `sagemaker` python SDK to deploy QwQ to Amazon SageMaker. We need to make sure to have an AWS account configured and the `sagemaker` python SDK installed.\n\nPython\n\n```python\n!pip install \"sagemaker>=2.232.2\" --upgrade --quiet\n \n```\n\nIf you are going to use Sagemaker in a local environment. You need access to an IAM Role with the required permissions for Sagemaker. You can find [here](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-roles.html) more about it.\n\nPython\n\n```python\nimport sagemaker\nimport boto3\nsess = sagemaker.Session()\n# sagemaker session bucket -> used for uploading data, models and logs\n# sagemaker will automatically create this bucket if it not exists\nsagemaker_session_bucket=None\nif sagemaker_session_bucket is None and sess is not None:\n # set to default bucket if a bucket name is not given\n sagemaker_session_bucket = sess.default_bucket()\n \ntry:\n role = sagemaker.get_execution_role()\nexcept ValueError:\n iam = boto3.client('iam')\n role = iam.get_role(RoleName='sagemaker_execution_role')['Role']['Arn']\n \nsess = sagemaker.Session(default_bucket=sagemaker_session_bucket)\n \nprint(f\"sagemaker role arn: {role}\")\nprint(f\"sagemaker session region: {sess.boto_region_name}\")\n \n```\n\n## 2\\. Retrieve the new Hugging Face LLM DLC\n\nCompared to deploying regular Hugging Face models we first need to retrieve the container uri and provide it to our `HuggingFaceModel` model class with a `image_uri` pointing to the image. To retrieve the new Hugging Face LLM DLC in Amazon SageMaker, we can use the `get_huggingface_llm_image_uri` method provided by the `sagemaker` SDK. This method allows us to retrieve the URI for the desired Hugging Face LLM DLC based on the specified `backend`, `session`, `region`, and `version`. You can find the available versions [here](https://github.com/aws/deep-learning-containers/blob/master/available_images.md#huggingface-text-generation-inference-containers)\n\nPython\n\n```python\n# TODO: COMMENT IN WHEN IMAGE IS RELEASED\n# from sagemaker.huggingface import get_huggingface_llm_image_uri\n \n# # retrieve the llm image uri\n# llm_image = get_huggingface_llm_image_uri(\n# \"huggingface\",\n# version=\"2.4.0\"\n# )\n \n# # print ecr image uri\n# print(f\"llm image uri: {llm_image}\")\n \nllm_image = f\"763104351884.dkr.ecr.{sess.boto_region_name}.amazonaws.com/huggingface-pytorch-tgi-inference:2.4-tgi2.3-gpu-py311-cu124-ubuntu22.04\"\n \n```\n\n## 3\\. Deploy QwQ-32B-Preview to Amazon SageMaker\n\nTo deploy [Qwen/QwQ-32B-Preview](https://huggingface.co/Qwen/QwQ-32B-Preview) to Amazon SageMaker we create a `HuggingFaceModel` model class and define our endpoint configuration including the `hf_model_id`, `instance_type` etc. We will use a `g6.12xlarge` instance type, which has 4 NVIDIA L4 GPUs and 96GB of GPU memory.\n\nQwQ-32B is a 32B parameter big dense decoder requiring ~64GB of raw GPU memory to load it + additional PyTorch overhead and storage for the KV-Cache Storage.\n\n| Model | Instance Type | \\# of GPUs per replica | quantization |\n| ------------------------------------------------------------------ | --------------------- | ---------------------- | ------------ |\n| [QwQ-32B-Preview](https://www.philschmid.de/Qwen/QwQ-32B-Preview) | `(ml.)g6e.2xlarge` | 1 | int4   |\n| [QwQ-32B-Preview](https://www.philschmid.de/Qwen/QwQ-32B-Preview) | `(ml.)g6e.2xlarge` | 1 | fp8   |\n| [QwQ-32B-Preview](https://www.philschmid.de/Qwen/QwQ-32B-Preview) | `(ml.)g5/g6.12xlarge` | 4 | \\-   |\n| [QwQ-32B-Preview](https://www.philschmid.de/Qwen/QwQ-32B-Preview) | `(ml.)g6e.12xlarge` | 4 | \\-   |\n| [QwQ-32B-Preview](https://www.philschmid.de/Qwen/QwQ-32B-Preview) | `(ml.)p4d.24xlarge` | 8 | \\-   |\n\nWe are going to use the `g6.12xlarge` instance type with 4 GPUs.\n\nPython\n\n```python\nimport json\nfrom sagemaker.huggingface import HuggingFaceModel\n \n# sagemaker config\ninstance_type = \"ml.g6.12xlarge\"\nnumber_of_gpu = 4\nhealth_check_timeout = 300\n \n# Define Model and Endpoint configuration parameter\nconfig = {\n 'HF_MODEL_ID': \"Qwen/QwQ-32B-Preview\", # model_id from hf.co/models\n 'SM_NUM_GPUS': json.dumps(number_of_gpu), # Number of GPU used per replica\n 'MAX_INPUT_LENGTH': json.dumps(4096), # Max length of input text\n 'MAX_TOTAL_TOKENS': json.dumps(8192), # Max length of the generation (including input text)\n 'HF_HUB_ENABLE_HF_TRANSFER': \"1\", # Enable HF transfer for faster downloads\n 'MESSAGES_API_ENABLED': \"true\", # Enable OpenAI compatible messages API\n}\n \n# create HuggingFaceModel with the image uri\nllm_model = HuggingFaceModel(\n role=role,\n image_uri=llm_image,\n env=config\n)\n```\n\nAfter we have created the `HuggingFaceModel` we can deploy it to Amazon SageMaker using the `deploy` method. We will deploy the model with the `ml.g6.12xlarge` instance type. TGI will automatically distribute and shard the model across all GPUs.\n\nPython\n\n```python\n# Deploy model to an endpoint\nllm = llm_model.deploy(\n initial_instance_count=1,\n instance_type=instance_type,\n container_startup_health_check_timeout=health_check_timeout, # 10 minutes to be able to load the model\n)\n \n```\n\nSageMaker will now create our endpoint and deploy the model to it. This can takes a 10-15 minutes.\n\n## 4\\. Run reasoning with QwQ and solve complex math problems\n\nAfter our endpoint is deployed we can run inference on it. We will use the `predict` method from the `predictor` to run inference on our endpoint. We create a helper [stream\\_request.py](https://www.philschmid.de/demo/stream_request.py) to stream tokens. This makes it easier to follow the reasoning process.\n\nQwQ is trained for advancing AI reasoning problems, like complex math problems using chain of thought similar to OpenAI's o1. We added small helper util to stream the response as it generates a lot of tokens to solve the problems.\n\nPython\n\n```python\nimport sys\nsys.path.append(\"../demo\") \nfrom stream_request import create_streamer\n \nstreamer = create_streamer(llm.endpoint_name)\n \nprompt = \"How many r in strawberry.\"\nres = streamer(prompt, max_tokens=4096)\n \nfor chunk in res:\n print(chunk, end=\"\", flush=True)\n```\n\n``\n\n`Let's see. The word is \"strawberry.\" I need to find out how many 'r's are in it. Okay, first, I'll spell it out slowly: s-t-r-a-w-b-e-r-r-y. Okay, now, I'll count the 'r's. Let's see: there's an 'r' after the 't', then another 'r' towards the end, and then another one after that. Wait, let's check again. s-t-r-a-w-b-e-r-r-y. So, the first 'r' is the third letter, then there's another 'r' before the last letter, and another one right after it. So, that's three 'r's in \"strawberry.\" But, maybe I should double-check because it's easy to miss letters when counting. Let me write it down: s-t-r-a-w-b-e-r-r-y. Now, I'll point to each 'r' one by one. First 'r' here, second 'r' here, and third 'r' here. Yep, three 'r's in \"strawberry.\" I think that's correct. [Final Answer] 3`\n\n## 5\\. Clean up\n\nTo clean up, we can delete the model and endpoint.\n", "dateCreated": "2026-09-27T19:29:15.293681927+00:00", "date_created": "2024-12-03T00:00:00+00:00", "date_modified": "2024-12-03T00:00:00+00:00", "date_published": "2024-12-03T00:00:00+00:00", "description": "Qwen's QwQ-32B-Preview is the best open reasoning model for mathematical and programming reasoning capabilities among open models directly competing with OpenAI o1.", "external_url": "https://www.philschmid.de/sagemaker-deploy-qwq", "git": { "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e", "path": "items/philschmid-de/2024/12/2024-12-03-deploy-qwq-32b-preview-the-best-open-reasoning-model-on-aws.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/philschmid-de/2024/12/2024-12-03-deploy-qwq-32b-preview-the-best-open-reasoning-model-on-aws.md" }, "id": "urn:aggr:item:e99d60d7d6eaaf3f6828dcc649df2c41003a3343", "isBasedOn": { "@id": "https://www.philschmid.de/sagemaker-deploy-qwq" }, "mainEntity": { "@id": "https://www.philschmid.de/sagemaker-deploy-qwq", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/philschmid-de/2024-12-03-deploy-qwq-32b-preview-the-best-open-reasoning-model-on-aws/#webpage" }, "dateModified": "2024-12-03T00:00:00+00:00", "datePublished": "2024-12-03T00:00:00+00:00", "headline": "Deploy QwQ-32B-Preview the best open Reasoning Model on AWS with Hugging Face", "timeRequired": "PT5M", "url": "https://www.philschmid.de/sagemaker-deploy-qwq", "wordCount": 1102 }, "name": "Archived snapshot: Deploy QwQ-32B-Preview the best open Reasoning Model on AWS with Hugging Face", "preview": null, "reading_minutes": 5, "replicated_at": null, "source": "philschmid-de", "source_name": "philschmid.de - RSS feed", "summary": "Qwen's QwQ-32B-Preview is the best open reasoning model for mathematical and programming reasoning capabilities among open models directly competing with OpenAI o1.", "tags": [], "temporalCoverage": "2026-09-27T19:29:15.293681927+00:00", "title": "Deploy QwQ-32B-Preview the best open Reasoning Model on AWS with Hugging Face", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2024-12-03-deploy-qwq-32b-preview-the-best-open-reasoning-model-on-aws/", "word_count": 1102 }