{ "@context": "https://schema.org", "@id": "https://aggr.aymericbeaumet.com/items/hnrss-org/2026-10-20-deepseek-just-open-sourced-its-inference-engine/#webpage", "@type": [ "WebPage", "ArchiveComponent" ], "articleBody": "DeepInfer is the high-performance inference framework that powers DeepSeek's large-scale production services. This open-source release provides offline inference with the core features needed for serving at scale.\n\nSupported models:\n\n- DeepSeek-V4.1-Flash\n\nSupported features:\n\n- Optimized kernels, model and sampler runtime\n- Parallelism: Attention DP / balanced CP, MoE EP + EPLB\n- DSpark and confidence-scheduled verification\n- SWA bounded replay\n- Asynchronous scheduling (zero-overhead batch scheduler)\n- Batch-invariant inference\n- Deterministic sampling\n- Structured outputs\n\nFeature details and configuration\n\n- **Optimized kernels:** [FlashMLA](https://github.com/deepseek-ai/FlashMLA), [DeepGEMM](https://github.com/deepseek-ai/DeepGEMM), [DeepEP](https://github.com/deepseek-ai/DeepEP), [TileKernels](https://github.com/deepseek-ai/TileKernels), [DeepSelect](https://github.com/deepseek-ai/DeepSelect), and custom [TileLang kernels](https://github.com/deepseek-ai/DeepInfer/blob/main/deepinfer/model_executor/layers/tilelang_ops).\n- **DSpark and confidence-scheduled verification:** accelerate decoding with draft tokens and use confidence estimates to schedule verification. Set `DEEPINFER_ENABLE_MTP_VARLEN=1`; `DEEPINFER_USE_NEXTN_DECODE` controls the maximum draft length (default: 5).\n- **Decoder SWA bounded replay:** reduce decoder prefill work by replaying only a bounded suffix. Set `DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact` and `DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140` for the default V4.1 settings; the retained suffix must cover the SWA window plus the maximum backtracking length.\n- **Asynchronous scheduling:** overlap CPU batch scheduling with device execution. Enabled by default.\n- **Batch-invariant inference:** keep model results identical across different batch compositions. Enable with `DEEPINFER_BATCH_INVARIANT=1` (NVIDIA only).\n- **Deterministic sampling:** produce identical outputs for identical input tokens and sampling parameters. Set `DEEPINFER_BATCH_INVARIANT=1 DEEPINFER_ENABLE_MTP_VARLEN=0` and pass a fixed `seq_seed` to `SamplingParams` (NVIDIA only).\n- **Structured outputs:** constrain generation to a regex, JSON schema, or EBNF grammar. Pass `so_args={\"schema_type\": ..., \"schema\": ...}` to `SamplingParams`; see [examples](https://github.com/deepseek-ai/DeepInfer/blob/main/examples/_structured_output.py).\n\nSupported hardware:\n\n- NVIDIA SM100 GPUs\n- Huawei Ascend 950 NPUs\n\nRequired for a complete inference service (not included in this release):\n\n- A KV cache system\n- VE/Prefill/Decode disaggregation (most required engine functionality is already implemented)\n- Load balancing across data-parallel replicas for both prefill and decode\n\n## Getting started\n\n### Prerequisites\n\n- Linux\n- GCC >= 13.0\n- Python 3.12\n- CUDA 13 (NVIDIA only)\n- CANN 9.2 (Ascend only)\n\n### Build from source\n\nBefore building DeepInfer, please create and activate an empty Python 3.12 environment.\n\n```shell\n# Use `ascend` instead for the Ascend platform\n# Keep it exported in the environment where you run DeepInfer\nexport DEEPINFER_ACCELERATOR=nvidia\n./build.sh\n```\n\n### Offline inference\n\nThe default checkpoint, [`deepseek-ai/DeepSeek-V4.1-Flash-1017`](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017), is downloaded into the Hugging Face cache on the first run; set `DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017` to run from a local copy instead.\n\n```shell\npython examples/offline_inference.py\n```\n\nSee [examples](https://github.com/deepseek-ai/DeepInfer/blob/main/examples/README.md) for details.\n\n## Performance benchmarks\n\n| Prefill | Decode |\n| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| [![Prefill: non-cached input throughput vs. latency](https://github.com/deepseek-ai/DeepInfer/raw/main/doc/images/prefill_pareto.png)](https://github.com/deepseek-ai/DeepInfer/blob/main/doc/images/prefill_pareto.png) | [![Decode: output throughput vs. per-user speed](https://github.com/deepseek-ai/DeepInfer/raw/main/doc/images/decode_pareto.png)](https://github.com/deepseek-ai/DeepInfer/blob/main/doc/images/decode_pareto.png) |\n\nMeasured on 4 SM100 GPUs for prefill and 64 for decode — each within one scale-up domain. To reproduce the benchmarks, see [run\\_sweep.sh](https://github.com/deepseek-ai/DeepInfer/blob/main/examples/sweep/run_sweep.sh).\n\n## Evaluation benchmarks\n\nSee [eval/README.md](https://github.com/deepseek-ai/DeepInfer/blob/main/eval/README.md).\n\n## License\n\nDeepInfer is released under the Apache License 2.0. See [LICENSE](https://github.com/deepseek-ai/DeepInfer/blob/main/LICENSE) for details.\n\n## Acknowledgments\n\nDeepInfer was originally built primarily on [vLLM](https://github.com/vllm-project/vllm), with additional inspiration from [SGLang](https://github.com/sgl-project/sglang) and [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM). We thank their developers.\n\n## Citation\n\n```text-bibtex\n@misc{deepinfer2026,\n title = {DeepInfer: High-performance, large-scale inference framework for DeepSeek models},\n author = {DeepSeek-AI},\n year = {2026},\n publisher = {GitHub},\n howpublished = {\\url{https://github.com/deepseek-ai/DeepInfer}},\n}\n```\n", "authors": [ "ggcr" ], "captured_at": "2026-10-20T12:56:01.525437407+00:00", "category": "news", "content_html": "

DeepInfer is the high-performance inference framework that powers DeepSeek's large-scale production services. This open-source release provides offline inference with the core features needed for serving at scale.

\n

Supported models:

\n\n

Supported features:

\n\n

Feature details and configuration

\n\n

Supported hardware:

\n\n

Required for a complete inference service (not included in this release):

\n\n

Getting started

\n

Prerequisites

\n\n

Build from source

\n

Before building DeepInfer, please create and activate an empty Python 3.12 environment.

\n
# Use `ascend` instead for the Ascend platform\n# Keep it exported in the environment where you run DeepInfer\nexport DEEPINFER_ACCELERATOR=nvidia\n./build.sh\n
\n

Offline inference

\n

The default checkpoint, deepseek-ai/DeepSeek-V4.1-Flash-1017, is downloaded into the Hugging Face cache on the first run; set DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017 to run from a local copy instead.

\n
python examples/offline_inference.py\n
\n

See examples for details.

\n

Performance benchmarks

\n\n\n\n\n\n\n\n\n\n\n\n\n\n
PrefillDecode
\"Prefill:\"Decode:
\n

Measured on 4 SM100 GPUs for prefill and 64 for decode — each within one scale-up domain. To reproduce the benchmarks, see run_sweep.sh.

\n

Evaluation benchmarks

\n

See eval/README.md.

\n

License

\n

DeepInfer is released under the Apache License 2.0. See LICENSE for details.

\n

Acknowledgments

\n

DeepInfer was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM. We thank their developers.

\n

Citation

\n
@misc{deepinfer2026,\n  title = {DeepInfer: High-performance, large-scale inference framework for DeepSeek models},\n  author = {DeepSeek-AI},\n  year = {2026},\n  publisher = {GitHub},\n  howpublished = {\\url{https://github.com/deepseek-ai/DeepInfer}},\n}\n
\n", "content_markdown": "DeepInfer is the high-performance inference framework that powers DeepSeek's large-scale production services. This open-source release provides offline inference with the core features needed for serving at scale.\n\nSupported models:\n\n- DeepSeek-V4.1-Flash\n\nSupported features:\n\n- Optimized kernels, model and sampler runtime\n- Parallelism: Attention DP / balanced CP, MoE EP + EPLB\n- DSpark and confidence-scheduled verification\n- SWA bounded replay\n- Asynchronous scheduling (zero-overhead batch scheduler)\n- Batch-invariant inference\n- Deterministic sampling\n- Structured outputs\n\nFeature details and configuration\n\n- **Optimized kernels:** [FlashMLA](https://github.com/deepseek-ai/FlashMLA), [DeepGEMM](https://github.com/deepseek-ai/DeepGEMM), [DeepEP](https://github.com/deepseek-ai/DeepEP), [TileKernels](https://github.com/deepseek-ai/TileKernels), [DeepSelect](https://github.com/deepseek-ai/DeepSelect), and custom [TileLang kernels](https://github.com/deepseek-ai/DeepInfer/blob/main/deepinfer/model_executor/layers/tilelang_ops).\n- **DSpark and confidence-scheduled verification:** accelerate decoding with draft tokens and use confidence estimates to schedule verification. Set `DEEPINFER_ENABLE_MTP_VARLEN=1`; `DEEPINFER_USE_NEXTN_DECODE` controls the maximum draft length (default: 5).\n- **Decoder SWA bounded replay:** reduce decoder prefill work by replaying only a bounded suffix. Set `DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact` and `DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140` for the default V4.1 settings; the retained suffix must cover the SWA window plus the maximum backtracking length.\n- **Asynchronous scheduling:** overlap CPU batch scheduling with device execution. Enabled by default.\n- **Batch-invariant inference:** keep model results identical across different batch compositions. Enable with `DEEPINFER_BATCH_INVARIANT=1` (NVIDIA only).\n- **Deterministic sampling:** produce identical outputs for identical input tokens and sampling parameters. Set `DEEPINFER_BATCH_INVARIANT=1 DEEPINFER_ENABLE_MTP_VARLEN=0` and pass a fixed `seq_seed` to `SamplingParams` (NVIDIA only).\n- **Structured outputs:** constrain generation to a regex, JSON schema, or EBNF grammar. Pass `so_args={\"schema_type\": ..., \"schema\": ...}` to `SamplingParams`; see [examples](https://github.com/deepseek-ai/DeepInfer/blob/main/examples/_structured_output.py).\n\nSupported hardware:\n\n- NVIDIA SM100 GPUs\n- Huawei Ascend 950 NPUs\n\nRequired for a complete inference service (not included in this release):\n\n- A KV cache system\n- VE/Prefill/Decode disaggregation (most required engine functionality is already implemented)\n- Load balancing across data-parallel replicas for both prefill and decode\n\n## Getting started\n\n### Prerequisites\n\n- Linux\n- GCC >= 13.0\n- Python 3.12\n- CUDA 13 (NVIDIA only)\n- CANN 9.2 (Ascend only)\n\n### Build from source\n\nBefore building DeepInfer, please create and activate an empty Python 3.12 environment.\n\n```shell\n# Use `ascend` instead for the Ascend platform\n# Keep it exported in the environment where you run DeepInfer\nexport DEEPINFER_ACCELERATOR=nvidia\n./build.sh\n```\n\n### Offline inference\n\nThe default checkpoint, [`deepseek-ai/DeepSeek-V4.1-Flash-1017`](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017), is downloaded into the Hugging Face cache on the first run; set `DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017` to run from a local copy instead.\n\n```shell\npython examples/offline_inference.py\n```\n\nSee [examples](https://github.com/deepseek-ai/DeepInfer/blob/main/examples/README.md) for details.\n\n## Performance benchmarks\n\n| Prefill | Decode |\n| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| [![Prefill: non-cached input throughput vs. latency](https://github.com/deepseek-ai/DeepInfer/raw/main/doc/images/prefill_pareto.png)](https://github.com/deepseek-ai/DeepInfer/blob/main/doc/images/prefill_pareto.png) | [![Decode: output throughput vs. per-user speed](https://github.com/deepseek-ai/DeepInfer/raw/main/doc/images/decode_pareto.png)](https://github.com/deepseek-ai/DeepInfer/blob/main/doc/images/decode_pareto.png) |\n\nMeasured on 4 SM100 GPUs for prefill and 64 for decode — each within one scale-up domain. To reproduce the benchmarks, see [run\\_sweep.sh](https://github.com/deepseek-ai/DeepInfer/blob/main/examples/sweep/run_sweep.sh).\n\n## Evaluation benchmarks\n\nSee [eval/README.md](https://github.com/deepseek-ai/DeepInfer/blob/main/eval/README.md).\n\n## License\n\nDeepInfer is released under the Apache License 2.0. See [LICENSE](https://github.com/deepseek-ai/DeepInfer/blob/main/LICENSE) for details.\n\n## Acknowledgments\n\nDeepInfer was originally built primarily on [vLLM](https://github.com/vllm-project/vllm), with additional inspiration from [SGLang](https://github.com/sgl-project/sglang) and [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM). We thank their developers.\n\n## Citation\n\n```text-bibtex\n@misc{deepinfer2026,\n title = {DeepInfer: High-performance, large-scale inference framework for DeepSeek models},\n author = {DeepSeek-AI},\n year = {2026},\n publisher = {GitHub},\n howpublished = {\\url{https://github.com/deepseek-ai/DeepInfer}},\n}\n```\n", "dateCreated": "2026-10-20T12:56:01.525437407+00:00", "date_created": "2026-10-20T09:08:54+00:00", "date_modified": "2026-10-20T09:08:54+00:00", "date_published": "2026-10-20T09:08:54+00:00", "description": "DeepInfer is the high-performance inference framework that powers DeepSeek's large-scale production services. This open-source release provides offline inference with the core features needed for serving at scale. Supported models:…", "external_url": "https://github.com/deepseek-ai/DeepInfer", "git": { "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e", "path": "items/hnrss-org/2026/10/2026-10-20-deepseek-just-open-sourced-its-inference-engine.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/hnrss-org/2026/10/2026-10-20-deepseek-just-open-sourced-its-inference-engine.md" }, "id": "urn:aggr:item:dfdf82203fa140acdefbe4452bc42c191a1168fc", "isBasedOn": { "@id": "https://github.com/deepseek-ai/DeepInfer" }, "mainEntity": { "@id": "https://github.com/deepseek-ai/DeepInfer", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/hnrss-org/2026-10-20-deepseek-just-open-sourced-its-inference-engine/#webpage" }, "author": [ { "@type": "Person", "name": "ggcr" } ], "dateModified": "2026-10-20T09:08:54+00:00", "datePublished": "2026-10-20T09:08:54+00:00", "headline": "DeepSeek just open-sourced its inference engine", "image": { "@type": "ImageObject", "caption": "DeepInfer: High-performance, large-scale inference framework for DeepSeek models - deepseek-ai/DeepInfer", "height": 90, "url": "https://opengraph.githubassets.com/2f1c2fc17da059a0f4782846a6f07a98efb17f74bbfdcd9195516247e979bcfc/deepseek-ai/DeepInfer", "width": 160 }, "timeRequired": "PT3M", "url": "https://github.com/deepseek-ai/DeepInfer", "wordCount": 463 }, "name": "Archived snapshot: DeepSeek just open-sourced its inference engine", "preview": { "alt": "DeepInfer: High-performance, large-scale inference framework for DeepSeek models - deepseek-ai/DeepInfer", "color": null, "height": 90, "placeholder": { "data_url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII=", "hash": "" }, "url": "https://opengraph.githubassets.com/2f1c2fc17da059a0f4782846a6f07a98efb17f74bbfdcd9195516247e979bcfc/deepseek-ai/DeepInfer", "width": 160 }, "reading_minutes": 3, "replicated_at": null, "source": "hnrss-org", "source_name": "Hacker News: Front Page", "summary": null, "tags": [], "temporalCoverage": "2026-10-20T12:56:01.525437407+00:00", "title": "DeepSeek just open-sourced its inference engine", "url": "https://aggr.aymericbeaumet.com/items/hnrss-org/2026-10-20-deepseek-just-open-sourced-its-inference-engine/", "word_count": 463 }