{ "@context": "https://schema.org", "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-06-26-run-a-vllm-server-on-hf-jobs-in-one-command/#webpage", "@type": [ "WebPage", "ArchiveComponent" ], "articleBody": "You can spin up a private, OpenAI-compatible LLM endpoint on Hugging Face infrastructure with a single command β€” no servers to provision, no Kubernetes, pay-per-second. Once it's up, you can query it from your laptop, a notebook, or anywhere else.\n\nIt's the quickest way to stand up a model for tests, evals, or batch generation. (If you're after a managed, production-ready service instead, that's what [Inference Endpoints](https://huggingface.co/docs/inference-endpoints) are for β€” [more on when to pick which](https://huggingface.co/blog/vllm-jobs#hf-jobs-or-inference-endpoints) at the end.)\n\nHere's the whole thing end to end.\n\n## Prerequisites\n\n- A payment method or a positive prepaid credit balance (Jobs is billed per‑minute by hardware usage).\n- `huggingface_hub >= 1.20.0`: `pip install -U \"huggingface_hub>=1.20.0\"`.\n- Logged in locally: `hf auth login`.\n\n## Launch the server\n\n`hf jobs run` is `docker run` for HF infrastructure. We use the official `vllm/vllm-openai` image, ask for a GPU with `--flavor`, and expose vLLM's port with `--expose`:\n\n```bash\nhf jobs run --flavor a10g-large --expose 8000 --timeout 2h \\\n vllm/vllm-openai:latest \\\n vllm serve Qwen/Qwen3-4B --host 0.0.0.0 --port 8000\n```\n\n`--expose 8000` routes the container's port through HF's public jobs proxy (see the [Serve Models guide](https://huggingface.co/docs/hub/jobs-serving) for the full reference). The command prints the URL your server is reachable at:\n\n```\nβœ“ Job started\n id: 6a381ca1953ed90bfb947332\n url: https://huggingface.co/jobs/qgallouedec/6a381ca1953ed90bfb947332\nHint: Exposed ports are reachable at (requires an HF token with read access to the job):\n https://6a381ca1953ed90bfb947332--8000.hf.jobs\n```\n\n`6a381ca1953ed90bfb947332` is your job ID. Keep track of it, we'll need it. We'll use `` as a placeholder for it in the rest of the post.\n\nGive it a couple of minutes to download weights and boot. When the logs show `Application startup complete`, you're live.\n\n## Query it from anywhere\n\nvLLM speaks the OpenAI API, and every request just needs your HF token as a bearer token. The quickest way to hit it is curl:\n\n```bash\ncurl https://--8000.hf.jobs/v1/chat/completions \\\n -H \"Authorization: Bearer $(hf auth token)\" \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"Qwen/Qwen3-4B\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hello!\"}],\n \"chat_template_kwargs\": {\"enable_thinking\": false}\n }'\n```\n\nwhich returns the usual OpenAI-style JSON, with `choices[0].message.content` holding `\"Hello! How can I assist you today? 😊\"`.\n\nOr, from Python, point the OpenAI client at the exposed URL and pass the token as the API key:\n\n```python\nfrom huggingface_hub import get_token\nfrom openai import OpenAI\n\nclient = OpenAI(\n base_url=\"https://--8000.hf.jobs/v1\",\n api_key=get_token(),\n)\nresp = client.chat.completions.create(\n model=\"Qwen/Qwen3-4B\",\n messages=[{\"role\": \"user\", \"content\": \"Hello!\"}],\n extra_body={\"chat_template_kwargs\": {\"enable_thinking\": False}},\n)\nprint(resp.choices[0].message.content)\n```\n\n```\nHello! How can I assist you today? 😊\n```\n\nQuick health check before you start: `curl https://--8000.hf.jobs/v1/models -H \"Authorization: Bearer $(hf auth token)\"` should list the model.\n\n> **πŸ” The endpoint is gated, not public.** Every request must carry an HF token with **read access to the job's namespace**. A plain browser visit will be rejected. In effect, the jobs proxy *is* your API gate: access is scoped to you (and your org). That's fine for private use, but treat the URL accordingly: don't share it expecting it to be open, and don't paste your token into untrusted places. If you need finer-grained or public access, put a proper gateway in front instead. Or see [HF Jobs or Inference Endpoints?](https://huggingface.co/blog/vllm-jobs#hf-jobs-or-inference-endpoints) below.\n\n## Clean up\n\nJobs are billed per second, so stop the server when you're done:\n\n```bash\nhf jobs cancel \n```\n\nThe `--timeout` you set is a safety net (it'll auto-stop), but cancelling explicitly is cheaper. An `a10g-large` runs at $1.50/hour β€” check `hf jobs hardware` for the full price list and pick the smallest flavor that fits your model.\n\n## Going further: bigger models\n\nThe same command scales to much larger models β€” pick a beefier `--flavor` and tell vLLM to shard the model across the GPUs with `--tensor-parallel-size`. For example, the 122B Qwen3.5 mixture-of-experts model on 2Γ— H200:\n\n```bash\nhf jobs run --flavor h200x2 --expose 8000 --timeout 2h \\\n vllm/vllm-openai:latest \\\n vllm serve Qwen/Qwen3.5-122B-A10B \\\n --host 0.0.0.0 --port 8000 --tensor-parallel-size 2 \\\n --max-model-len 32768 --max-num-seqs 256\n```\n\n`--tensor-parallel-size` should match the number of GPUs in the flavor (`h200x2` β†’ 2, `h200x8` β†’ 8). Run `hf jobs hardware` to see what's available and give bigger models a longer `--timeout`, since they take longer to download and load. For large models, H200 flavors are usually the best value.\n\nThe `--max-model-len 32768 --max-num-seqs 256` flags are specific to this model: Qwen3.5-122B is a hybrid Mamba/attention architecture with a 256K-token default context, which doesn't leave enough memory for vLLM's default batch settings. Capping the context length and concurrent-sequence count keeps it within the GPUs' memory. If a model fails to start with an out-of-memory or cache-block error, dialing these two down is the first thing to try. Everything else (the exposed URL, the OpenAI client, the token auth) stays exactly the same.\n\n## Going further: Chat with it in a UI\n\nPrefer a chat window over curl? A few lines of [Gradio](https://www.gradio.app/) point at the same endpoint. Add `--reasoning-parser deepseek_r1` to the `vllm serve` command so Qwen3's thinking comes back as a separate field (not necessary, but helpful), then run this code locally (you'll just need the job ID):\n\n```python\nimport gradio as gr\nfrom gradio import ChatMessage\nfrom huggingface_hub import get_token\nfrom openai import OpenAI\n\nclient = OpenAI(base_url=\"https://--8000.hf.jobs/v1\", api_key=get_token())\n\ndef chat(message, history):\n messages = [{\"role\": m[\"role\"], \"content\": m[\"content\"]} for m in history if not m.get(\"metadata\")]\n messages.append({\"role\": \"user\", \"content\": message})\n stream = client.chat.completions.create(model=\"Qwen/Qwen3-4B\", messages=messages, stream=True)\n\n thinking, answer = \"\", \"\"\n for chunk in stream:\n delta = chunk.choices[0].delta\n thinking += delta.model_extra.get(\"reasoning\", \"\")\n answer += delta.content or \"\"\n out = []\n if thinking.strip():\n status = \"done\" if answer.strip() else \"pending\"\n out.append(ChatMessage(role=\"assistant\", content=thinking, metadata={\"title\": \"πŸ’­ Thinking\", \"status\": status}))\n if answer.strip():\n out.append(ChatMessage(role=\"assistant\", content=answer))\n yield out\n\ngr.ChatInterface(chat).launch()\n```\n\nRun it, open `http://127.0.0.1:7860`, and chat β€” reasoning streams into the collapsible panel, the answer below.\n\n## Going further: SSH into the running server\n\nNeed to debug a startup failure, watch GPU memory, or tail logs interactively? You can open a shell straight into the running job. Launch it with `--ssh` and make sure your public key is registered at [huggingface.co/settings/keys](https://huggingface.co/settings/keys):\n\n```bash\nhf jobs run --flavor a10g-large --expose 8000 --timeout 2h --ssh \\\n vllm/vllm-openai:latest \\\n vllm serve Qwen/Qwen3-4B --host 0.0.0.0 --port 8000\n```\n\nthen connect with the job ID:\n\n```bash\nhf jobs ssh \n```\n\nYou're now inside the container, where you can run `nvidia-smi`, inspect the process, or poke at the model directly β€” which makes debugging and monitoring much easier than reading logs from the outside. SSH support requires `huggingface_hub >= 1.20.0`.\n\n## Going further: Use it as a coding-agent backend with Pi\n\nThe same endpoint can back a terminal coding agent. [Pi](https://pi.dev) is a provider-agnostic agent harness. Point it at the job and you get a Read/Write/Edit/Bash agent running on your own self-hosted model.\n\nOne thing to set up first: agents drive the model through tool calls, and vLLM only accepts those if the server is launched with tool calling enabled. So relaunch with `--enable-auto-tool-choice` and a `--tool-call-parser` matching the model family (`hermes` for Qwen3). Agents also benefit from a stronger model, so this is a good place to bring in the bigger one:\n\n```bash\nhf jobs run --flavor h200x2 --expose 8000 --timeout 2h \\\n vllm/vllm-openai:latest \\\n vllm serve Qwen/Qwen3.5-122B-A10B \\\n --host 0.0.0.0 --port 8000 --tensor-parallel-size 2 \\\n --max-model-len 32768 --max-num-seqs 256 \\\n --reasoning-parser deepseek_r1 \\\n --enable-auto-tool-choice --tool-call-parser hermes\n```\n\nThen add the job as a custom provider in `~/.pi/agent/models.json`:\n\n```json\n{\n \"providers\": {\n \"hf-jobs\": {\n \"baseUrl\": \"https://--8000.hf.jobs/v1\",\n \"api\": \"openai-completions\",\n \"apiKey\": \"!hf auth token\",\n \"models\": [\n { \"id\": \"Qwen/Qwen3.5-122B-A10B\" }\n ]\n }\n }\n}\n```\n\nThen launch the agent against it:\n\n```bash\npi\n```\n\nThe model you spun up a couple of commands ago, now driving an interactive coding agent in your terminal.\n\n## HF Jobs or Inference Endpoints?\n\nHF Jobs isn't the only way to serve a model on Hugging Face. [Inference Endpoints](https://huggingface.co/docs/inference-endpoints) are our managed product for the same job, and which one fits depends on what you're after.\n\nReach for **HF Jobs** when you want maximum flexibility and control: it's just `docker run` on HF infrastructure, so you pick the image, the exact `vllm serve` flags, and the hardware, and you pay per second for as long as the job runs. That makes it a great fit for experiments, one-off evals, batch generation, or kicking the tires on a model before committing to anything.\n\nReach for **Inference Endpoints** when you want something more production-ready. They add the operational niceties a long-lived service needs: finer-grained access control (an endpoint can be public, protected, or private), and scale-to-zero, so you're not billed during periods of inactivity. If you're standing up a durable endpoint rather than running a job, that's the tool to grab.\n\n## Further reading\n\nThis post sticks to vLLM, but the same expose-a-port pattern works with any OpenAI-compatible server. To serve GGUFs with llama.cpp or run SGLang instead, see the [Serve Models on Jobs guide](https://huggingface.co/docs/hub/jobs-serving), which walks through those backends.\n", "authors": [], "captured_at": "2026-10-19T01:26:53.998049939+00:00", "category": "ai", "content_html": "

You can spin up a private, OpenAI-compatible LLM endpoint on Hugging Face infrastructure with a single command β€” no servers to provision, no Kubernetes, pay-per-second. Once it's up, you can query it from your laptop, a notebook, or anywhere else.

\n

It's the quickest way to stand up a model for tests, evals, or batch generation. (If you're after a managed, production-ready service instead, that's what Inference Endpoints are for β€” more on when to pick which at the end.)

\n

Here's the whole thing end to end.

\n

Prerequisites

\n
    \n
  • A payment method or a positive prepaid credit balance (Jobs is billed per‑minute by hardware usage).
    • \n
  • huggingface_hub >= 1.20.0: pip install -U "huggingface_hub>=1.20.0".
    • \n
  • Logged in locally: hf auth login.
    • \n
\n

Launch the server

\n

hf jobs run is docker run for HF infrastructure. We use the official vllm/vllm-openai image, ask for a GPU with --flavor, and expose vLLM's port with --expose:

\n
hf jobs run --flavor a10g-large --expose 8000 --timeout 2h \\\n  vllm/vllm-openai:latest \\\n  vllm serve Qwen/Qwen3-4B --host 0.0.0.0 --port 8000\n
\n

--expose 8000 routes the container's port through HF's public jobs proxy (see the Serve Models guide for the full reference). The command prints the URL your server is reachable at:

\n
βœ“ Job started\n  id: 6a381ca1953ed90bfb947332\n  url: https://huggingface.co/jobs/qgallouedec/6a381ca1953ed90bfb947332\nHint: Exposed ports are reachable at (requires an HF token with read access to the job):\n  https://6a381ca1953ed90bfb947332--8000.hf.jobs\n
\n

6a381ca1953ed90bfb947332 is your job ID. Keep track of it, we'll need it. We'll use <job_id> as a placeholder for it in the rest of the post.

\n

Give it a couple of minutes to download weights and boot. When the logs show Application startup complete, you're live.

\n

Query it from anywhere

\n

vLLM speaks the OpenAI API, and every request just needs your HF token as a bearer token. The quickest way to hit it is curl:

\n
curl https://<job_id>--8000.hf.jobs/v1/chat/completions \\\n  -H "Authorization: Bearer $(hf auth token)" \\\n  -H "Content-Type: application/json" \\\n  -d '{\n    "model": "Qwen/Qwen3-4B",\n    "messages": [{"role": "user", "content": "Hello!"}],\n    "chat_template_kwargs": {"enable_thinking": false}\n  }'\n
\n

which returns the usual OpenAI-style JSON, with choices[0].message.content holding "Hello! How can I assist you today? 😊".

\n

Or, from Python, point the OpenAI client at the exposed URL and pass the token as the API key:

\n
from huggingface_hub import get_token\nfrom openai import OpenAI\n\nclient = OpenAI(\n    base_url="https://<job_id>--8000.hf.jobs/v1",\n    api_key=get_token(),\n)\nresp = client.chat.completions.create(\n    model="Qwen/Qwen3-4B",\n    messages=[{"role": "user", "content": "Hello!"}],\n    extra_body={"chat_template_kwargs": {"enable_thinking": False}},\n)\nprint(resp.choices[0].message.content)\n
\n
Hello! How can I assist you today? 😊\n
\n

Quick health check before you start: curl https://<job_id>--8000.hf.jobs/v1/models -H "Authorization: Bearer $(hf auth token)" should list the model.

\n
\n

πŸ” The endpoint is gated, not public. Every request must carry an HF token with read access to the job's namespace. A plain browser visit will be rejected. In effect, the jobs proxy is your API gate: access is scoped to you (and your org). That's fine for private use, but treat the URL accordingly: don't share it expecting it to be open, and don't paste your token into untrusted places. If you need finer-grained or public access, put a proper gateway in front instead. Or see HF Jobs or Inference Endpoints? below.

\n
\n

Clean up

\n

Jobs are billed per second, so stop the server when you're done:

\n
hf jobs cancel <job_id>\n
\n

The --timeout you set is a safety net (it'll auto-stop), but cancelling explicitly is cheaper. An a10g-large runs at $1.50/hour β€” check hf jobs hardware for the full price list and pick the smallest flavor that fits your model.

\n

Going further: bigger models

\n

The same command scales to much larger models β€” pick a beefier --flavor and tell vLLM to shard the model across the GPUs with --tensor-parallel-size. For example, the 122B Qwen3.5 mixture-of-experts model on 2Γ— H200:

\n
hf jobs run --flavor h200x2 --expose 8000 --timeout 2h \\\n  vllm/vllm-openai:latest \\\n  vllm serve Qwen/Qwen3.5-122B-A10B \\\n  --host 0.0.0.0 --port 8000 --tensor-parallel-size 2 \\\n  --max-model-len 32768 --max-num-seqs 256\n
\n

--tensor-parallel-size should match the number of GPUs in the flavor (h200x2 β†’ 2, h200x8 β†’ 8). Run hf jobs hardware to see what's available and give bigger models a longer --timeout, since they take longer to download and load. For large models, H200 flavors are usually the best value.

\n

The --max-model-len 32768 --max-num-seqs 256 flags are specific to this model: Qwen3.5-122B is a hybrid Mamba/attention architecture with a 256K-token default context, which doesn't leave enough memory for vLLM's default batch settings. Capping the context length and concurrent-sequence count keeps it within the GPUs' memory. If a model fails to start with an out-of-memory or cache-block error, dialing these two down is the first thing to try. Everything else (the exposed URL, the OpenAI client, the token auth) stays exactly the same.

\n

Going further: Chat with it in a UI

\n

Prefer a chat window over curl? A few lines of Gradio point at the same endpoint. Add --reasoning-parser deepseek_r1 to the vllm serve command so Qwen3's thinking comes back as a separate field (not necessary, but helpful), then run this code locally (you'll just need the job ID):

\n
import gradio as gr\nfrom gradio import ChatMessage\nfrom huggingface_hub import get_token\nfrom openai import OpenAI\n\nclient = OpenAI(base_url="https://<job_id>--8000.hf.jobs/v1", api_key=get_token())\n\ndef chat(message, history):\n    messages = [{"role": m["role"], "content": m["content"]} for m in history if not m.get("metadata")]\n    messages.append({"role": "user", "content": message})\n    stream = client.chat.completions.create(model="Qwen/Qwen3-4B", messages=messages, stream=True)\n\n    thinking, answer = "", ""\n    for chunk in stream:\n        delta = chunk.choices[0].delta\n        thinking += delta.model_extra.get("reasoning", "")\n        answer += delta.content or ""\n        out = []\n        if thinking.strip():\n            status = "done" if answer.strip() else "pending"\n            out.append(ChatMessage(role="assistant", content=thinking, metadata={"title": "πŸ’­ Thinking", "status": status}))\n        if answer.strip():\n            out.append(ChatMessage(role="assistant", content=answer))\n        yield out\n\ngr.ChatInterface(chat).launch()\n
\n

Run it, open http://127.0.0.1:7860, and chat β€” reasoning streams into the collapsible panel, the answer below.

\n

Going further: SSH into the running server

\n

Need to debug a startup failure, watch GPU memory, or tail logs interactively? You can open a shell straight into the running job. Launch it with --ssh and make sure your public key is registered at huggingface.co/settings/keys:

\n
hf jobs run --flavor a10g-large --expose 8000 --timeout 2h --ssh \\\n  vllm/vllm-openai:latest \\\n  vllm serve Qwen/Qwen3-4B --host 0.0.0.0 --port 8000\n
\n

then connect with the job ID:

\n
hf jobs ssh <job_id>\n
\n

You're now inside the container, where you can run nvidia-smi, inspect the process, or poke at the model directly β€” which makes debugging and monitoring much easier than reading logs from the outside. SSH support requires huggingface_hub >= 1.20.0.

\n

Going further: Use it as a coding-agent backend with Pi

\n

The same endpoint can back a terminal coding agent. Pi is a provider-agnostic agent harness. Point it at the job and you get a Read/Write/Edit/Bash agent running on your own self-hosted model.

\n

One thing to set up first: agents drive the model through tool calls, and vLLM only accepts those if the server is launched with tool calling enabled. So relaunch with --enable-auto-tool-choice and a --tool-call-parser matching the model family (hermes for Qwen3). Agents also benefit from a stronger model, so this is a good place to bring in the bigger one:

\n
hf jobs run --flavor h200x2 --expose 8000 --timeout 2h \\\n  vllm/vllm-openai:latest \\\n  vllm serve Qwen/Qwen3.5-122B-A10B \\\n  --host 0.0.0.0 --port 8000 --tensor-parallel-size 2 \\\n  --max-model-len 32768 --max-num-seqs 256 \\\n  --reasoning-parser deepseek_r1 \\\n  --enable-auto-tool-choice --tool-call-parser hermes\n
\n

Then add the job as a custom provider in ~/.pi/agent/models.json:

\n
{\n  "providers": {\n    "hf-jobs": {\n      "baseUrl": "https://<job_id>--8000.hf.jobs/v1",\n      "api": "openai-completions",\n      "apiKey": "!hf auth token",\n      "models": [\n        { "id": "Qwen/Qwen3.5-122B-A10B" }\n      ]\n    }\n  }\n}\n
\n

Then launch the agent against it:

\n
pi\n
\n

The model you spun up a couple of commands ago, now driving an interactive coding agent in your terminal.

\n

HF Jobs or Inference Endpoints?

\n

HF Jobs isn't the only way to serve a model on Hugging Face. Inference Endpoints are our managed product for the same job, and which one fits depends on what you're after.

\n

Reach for HF Jobs when you want maximum flexibility and control: it's just docker run on HF infrastructure, so you pick the image, the exact vllm serve flags, and the hardware, and you pay per second for as long as the job runs. That makes it a great fit for experiments, one-off evals, batch generation, or kicking the tires on a model before committing to anything.

\n

Reach for Inference Endpoints when you want something more production-ready. They add the operational niceties a long-lived service needs: finer-grained access control (an endpoint can be public, protected, or private), and scale-to-zero, so you're not billed during periods of inactivity. If you're standing up a durable endpoint rather than running a job, that's the tool to grab.

\n

Further reading

\n

This post sticks to vLLM, but the same expose-a-port pattern works with any OpenAI-compatible server. To serve GGUFs with llama.cpp or run SGLang instead, see the Serve Models on Jobs guide, which walks through those backends.

\n", "content_markdown": "You can spin up a private, OpenAI-compatible LLM endpoint on Hugging Face infrastructure with a single command β€” no servers to provision, no Kubernetes, pay-per-second. Once it's up, you can query it from your laptop, a notebook, or anywhere else.\n\nIt's the quickest way to stand up a model for tests, evals, or batch generation. (If you're after a managed, production-ready service instead, that's what [Inference Endpoints](https://huggingface.co/docs/inference-endpoints) are for β€” [more on when to pick which](https://huggingface.co/blog/vllm-jobs#hf-jobs-or-inference-endpoints) at the end.)\n\nHere's the whole thing end to end.\n\n## Prerequisites\n\n- A payment method or a positive prepaid credit balance (Jobs is billed per‑minute by hardware usage).\n- `huggingface_hub >= 1.20.0`: `pip install -U \"huggingface_hub>=1.20.0\"`.\n- Logged in locally: `hf auth login`.\n\n## Launch the server\n\n`hf jobs run` is `docker run` for HF infrastructure. We use the official `vllm/vllm-openai` image, ask for a GPU with `--flavor`, and expose vLLM's port with `--expose`:\n\n```bash\nhf jobs run --flavor a10g-large --expose 8000 --timeout 2h \\\n vllm/vllm-openai:latest \\\n vllm serve Qwen/Qwen3-4B --host 0.0.0.0 --port 8000\n```\n\n`--expose 8000` routes the container's port through HF's public jobs proxy (see the [Serve Models guide](https://huggingface.co/docs/hub/jobs-serving) for the full reference). The command prints the URL your server is reachable at:\n\n```\nβœ“ Job started\n id: 6a381ca1953ed90bfb947332\n url: https://huggingface.co/jobs/qgallouedec/6a381ca1953ed90bfb947332\nHint: Exposed ports are reachable at (requires an HF token with read access to the job):\n https://6a381ca1953ed90bfb947332--8000.hf.jobs\n```\n\n`6a381ca1953ed90bfb947332` is your job ID. Keep track of it, we'll need it. We'll use `` as a placeholder for it in the rest of the post.\n\nGive it a couple of minutes to download weights and boot. When the logs show `Application startup complete`, you're live.\n\n## Query it from anywhere\n\nvLLM speaks the OpenAI API, and every request just needs your HF token as a bearer token. The quickest way to hit it is curl:\n\n```bash\ncurl https://--8000.hf.jobs/v1/chat/completions \\\n -H \"Authorization: Bearer $(hf auth token)\" \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"Qwen/Qwen3-4B\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hello!\"}],\n \"chat_template_kwargs\": {\"enable_thinking\": false}\n }'\n```\n\nwhich returns the usual OpenAI-style JSON, with `choices[0].message.content` holding `\"Hello! How can I assist you today? 😊\"`.\n\nOr, from Python, point the OpenAI client at the exposed URL and pass the token as the API key:\n\n```python\nfrom huggingface_hub import get_token\nfrom openai import OpenAI\n\nclient = OpenAI(\n base_url=\"https://--8000.hf.jobs/v1\",\n api_key=get_token(),\n)\nresp = client.chat.completions.create(\n model=\"Qwen/Qwen3-4B\",\n messages=[{\"role\": \"user\", \"content\": \"Hello!\"}],\n extra_body={\"chat_template_kwargs\": {\"enable_thinking\": False}},\n)\nprint(resp.choices[0].message.content)\n```\n\n```\nHello! How can I assist you today? 😊\n```\n\nQuick health check before you start: `curl https://--8000.hf.jobs/v1/models -H \"Authorization: Bearer $(hf auth token)\"` should list the model.\n\n> **πŸ” The endpoint is gated, not public.** Every request must carry an HF token with **read access to the job's namespace**. A plain browser visit will be rejected. In effect, the jobs proxy *is* your API gate: access is scoped to you (and your org). That's fine for private use, but treat the URL accordingly: don't share it expecting it to be open, and don't paste your token into untrusted places. If you need finer-grained or public access, put a proper gateway in front instead. Or see [HF Jobs or Inference Endpoints?](https://huggingface.co/blog/vllm-jobs#hf-jobs-or-inference-endpoints) below.\n\n## Clean up\n\nJobs are billed per second, so stop the server when you're done:\n\n```bash\nhf jobs cancel \n```\n\nThe `--timeout` you set is a safety net (it'll auto-stop), but cancelling explicitly is cheaper. An `a10g-large` runs at $1.50/hour β€” check `hf jobs hardware` for the full price list and pick the smallest flavor that fits your model.\n\n## Going further: bigger models\n\nThe same command scales to much larger models β€” pick a beefier `--flavor` and tell vLLM to shard the model across the GPUs with `--tensor-parallel-size`. For example, the 122B Qwen3.5 mixture-of-experts model on 2Γ— H200:\n\n```bash\nhf jobs run --flavor h200x2 --expose 8000 --timeout 2h \\\n vllm/vllm-openai:latest \\\n vllm serve Qwen/Qwen3.5-122B-A10B \\\n --host 0.0.0.0 --port 8000 --tensor-parallel-size 2 \\\n --max-model-len 32768 --max-num-seqs 256\n```\n\n`--tensor-parallel-size` should match the number of GPUs in the flavor (`h200x2` β†’ 2, `h200x8` β†’ 8). Run `hf jobs hardware` to see what's available and give bigger models a longer `--timeout`, since they take longer to download and load. For large models, H200 flavors are usually the best value.\n\nThe `--max-model-len 32768 --max-num-seqs 256` flags are specific to this model: Qwen3.5-122B is a hybrid Mamba/attention architecture with a 256K-token default context, which doesn't leave enough memory for vLLM's default batch settings. Capping the context length and concurrent-sequence count keeps it within the GPUs' memory. If a model fails to start with an out-of-memory or cache-block error, dialing these two down is the first thing to try. Everything else (the exposed URL, the OpenAI client, the token auth) stays exactly the same.\n\n## Going further: Chat with it in a UI\n\nPrefer a chat window over curl? A few lines of [Gradio](https://www.gradio.app/) point at the same endpoint. Add `--reasoning-parser deepseek_r1` to the `vllm serve` command so Qwen3's thinking comes back as a separate field (not necessary, but helpful), then run this code locally (you'll just need the job ID):\n\n```python\nimport gradio as gr\nfrom gradio import ChatMessage\nfrom huggingface_hub import get_token\nfrom openai import OpenAI\n\nclient = OpenAI(base_url=\"https://--8000.hf.jobs/v1\", api_key=get_token())\n\ndef chat(message, history):\n messages = [{\"role\": m[\"role\"], \"content\": m[\"content\"]} for m in history if not m.get(\"metadata\")]\n messages.append({\"role\": \"user\", \"content\": message})\n stream = client.chat.completions.create(model=\"Qwen/Qwen3-4B\", messages=messages, stream=True)\n\n thinking, answer = \"\", \"\"\n for chunk in stream:\n delta = chunk.choices[0].delta\n thinking += delta.model_extra.get(\"reasoning\", \"\")\n answer += delta.content or \"\"\n out = []\n if thinking.strip():\n status = \"done\" if answer.strip() else \"pending\"\n out.append(ChatMessage(role=\"assistant\", content=thinking, metadata={\"title\": \"πŸ’­ Thinking\", \"status\": status}))\n if answer.strip():\n out.append(ChatMessage(role=\"assistant\", content=answer))\n yield out\n\ngr.ChatInterface(chat).launch()\n```\n\nRun it, open `http://127.0.0.1:7860`, and chat β€” reasoning streams into the collapsible panel, the answer below.\n\n## Going further: SSH into the running server\n\nNeed to debug a startup failure, watch GPU memory, or tail logs interactively? You can open a shell straight into the running job. Launch it with `--ssh` and make sure your public key is registered at [huggingface.co/settings/keys](https://huggingface.co/settings/keys):\n\n```bash\nhf jobs run --flavor a10g-large --expose 8000 --timeout 2h --ssh \\\n vllm/vllm-openai:latest \\\n vllm serve Qwen/Qwen3-4B --host 0.0.0.0 --port 8000\n```\n\nthen connect with the job ID:\n\n```bash\nhf jobs ssh \n```\n\nYou're now inside the container, where you can run `nvidia-smi`, inspect the process, or poke at the model directly β€” which makes debugging and monitoring much easier than reading logs from the outside. SSH support requires `huggingface_hub >= 1.20.0`.\n\n## Going further: Use it as a coding-agent backend with Pi\n\nThe same endpoint can back a terminal coding agent. [Pi](https://pi.dev) is a provider-agnostic agent harness. Point it at the job and you get a Read/Write/Edit/Bash agent running on your own self-hosted model.\n\nOne thing to set up first: agents drive the model through tool calls, and vLLM only accepts those if the server is launched with tool calling enabled. So relaunch with `--enable-auto-tool-choice` and a `--tool-call-parser` matching the model family (`hermes` for Qwen3). Agents also benefit from a stronger model, so this is a good place to bring in the bigger one:\n\n```bash\nhf jobs run --flavor h200x2 --expose 8000 --timeout 2h \\\n vllm/vllm-openai:latest \\\n vllm serve Qwen/Qwen3.5-122B-A10B \\\n --host 0.0.0.0 --port 8000 --tensor-parallel-size 2 \\\n --max-model-len 32768 --max-num-seqs 256 \\\n --reasoning-parser deepseek_r1 \\\n --enable-auto-tool-choice --tool-call-parser hermes\n```\n\nThen add the job as a custom provider in `~/.pi/agent/models.json`:\n\n```json\n{\n \"providers\": {\n \"hf-jobs\": {\n \"baseUrl\": \"https://--8000.hf.jobs/v1\",\n \"api\": \"openai-completions\",\n \"apiKey\": \"!hf auth token\",\n \"models\": [\n { \"id\": \"Qwen/Qwen3.5-122B-A10B\" }\n ]\n }\n }\n}\n```\n\nThen launch the agent against it:\n\n```bash\npi\n```\n\nThe model you spun up a couple of commands ago, now driving an interactive coding agent in your terminal.\n\n## HF Jobs or Inference Endpoints?\n\nHF Jobs isn't the only way to serve a model on Hugging Face. [Inference Endpoints](https://huggingface.co/docs/inference-endpoints) are our managed product for the same job, and which one fits depends on what you're after.\n\nReach for **HF Jobs** when you want maximum flexibility and control: it's just `docker run` on HF infrastructure, so you pick the image, the exact `vllm serve` flags, and the hardware, and you pay per second for as long as the job runs. That makes it a great fit for experiments, one-off evals, batch generation, or kicking the tires on a model before committing to anything.\n\nReach for **Inference Endpoints** when you want something more production-ready. They add the operational niceties a long-lived service needs: finer-grained access control (an endpoint can be public, protected, or private), and scale-to-zero, so you're not billed during periods of inactivity. If you're standing up a durable endpoint rather than running a job, that's the tool to grab.\n\n## Further reading\n\nThis post sticks to vLLM, but the same expose-a-port pattern works with any OpenAI-compatible server. To serve GGUFs with llama.cpp or run SGLang instead, see the [Serve Models on Jobs guide](https://huggingface.co/docs/hub/jobs-serving), which walks through those backends.\n", "dateCreated": "2026-10-19T01:26:53.998049939+00:00", "date_created": "2026-06-26T00:00:00+00:00", "date_modified": "2026-06-26T00:00:00+00:00", "date_published": "2026-06-26T00:00:00+00:00", "description": "You can spin up a private, OpenAI-compatible LLM endpoint on Hugging Face infrastructure with a single command β€” no servers to provision, no Kubernetes, pay-per-second. Once it's up, you can query it from your laptop, a notebook, or…", "external_url": "https://huggingface.co/blog/vllm-jobs", "git": { "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e", "path": "items/huggingface-co/2026/06/2026-06-26-run-a-vllm-server-on-hf-jobs-in-one-command.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/huggingface-co/2026/06/2026-06-26-run-a-vllm-server-on-hf-jobs-in-one-command.md" }, "id": "urn:aggr:item:4afc8df857fa9c76c4cf7d7be3962bafdebbabbc", "isBasedOn": { "@id": "https://huggingface.co/blog/vllm-jobs" }, "mainEntity": { "@id": "https://huggingface.co/blog/vllm-jobs", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-06-26-run-a-vllm-server-on-hf-jobs-in-one-command/#webpage" }, "dateModified": "2026-06-26T00:00:00+00:00", "datePublished": "2026-06-26T00:00:00+00:00", "headline": "Run a vLLM Server on HF Jobs in One Command", "image": { "@type": "ImageObject", "caption": null, "height": 90, "url": "https://huggingface.co/blog/assets/vllm-jobs/thumbnail.png", "width": 160 }, "timeRequired": "PT7M", "url": "https://huggingface.co/blog/vllm-jobs", "wordCount": 1541 }, "name": "Archived snapshot: Run a vLLM Server on HF Jobs in One Command", "preview": { "alt": null, "color": null, "height": 90, "placeholder": { "data_url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII=", "hash": "" }, "url": "https://huggingface.co/blog/assets/vllm-jobs/thumbnail.png", "width": 160 }, "reading_minutes": 7, "replicated_at": null, "source": "huggingface-co", "source_name": "Hugging Face - Blog", "summary": null, "tags": [], "temporalCoverage": "2026-10-19T01:26:53.998049939+00:00", "title": "Run a vLLM Server on HF Jobs in One Command", "url": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-06-26-run-a-vllm-server-on-hf-jobs-in-one-command/", "word_count": 1541 }