{
  "description": "Browse aymeric's, an independent, searchable archive of readable snapshots from followed feeds, preserved in Git with aggr.",
  "feed_url": "https://aggr.aymericbeaumet.com/sources/github.com/deepseek-ai/feed.json",
  "home_page_url": "https://aggr.aymericbeaumet.com/sources/github.com/deepseek-ai/",
  "items": [
    {
      "authors": [
        {
          "name": "ggcr"
        }
      ],
      "content_html": "<p>DeepInfer is the high-performance inference framework that powers DeepSeek's large-scale production services. This open-source release provides offline inference with the core features needed for serving at scale.</p>\n<p>Supported models:</p>\n<ul>\n<li>DeepSeek-V4.1-Flash</li>\n</ul>\n<p>Supported features:</p>\n<ul>\n<li>Optimized kernels, model and sampler runtime</li>\n<li>Parallelism: Attention DP / balanced CP, MoE EP + EPLB</li>\n<li>DSpark and confidence-scheduled verification</li>\n<li>SWA bounded replay</li>\n<li>Asynchronous scheduling (zero-overhead batch scheduler)</li>\n<li>Batch-invariant inference</li>\n<li>Deterministic sampling</li>\n<li>Structured outputs</li>\n</ul>\n<p>Feature details and configuration</p>\n<ul>\n<li><strong>Optimized kernels:</strong> <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://github.com/deepseek-ai/FlashMLA\">FlashMLA</a>, <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://github.com/deepseek-ai/DeepGEMM\">DeepGEMM</a>, <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://github.com/deepseek-ai/DeepEP\">DeepEP</a>, <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://github.com/deepseek-ai/TileKernels\">TileKernels</a>, <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://github.com/deepseek-ai/DeepSelect\">DeepSelect</a>, and custom <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://github.com/deepseek-ai/DeepInfer/blob/main/deepinfer/model_executor/layers/tilelang_ops\">TileLang kernels</a>.</li>\n<li><strong>DSpark and confidence-scheduled verification:</strong> accelerate decoding with draft tokens and use confidence estimates to schedule verification. Set <code>DEEPINFER_ENABLE_MTP_VARLEN=1</code>; <code>DEEPINFER_USE_NEXTN_DECODE</code> controls the maximum draft length (default: 5).</li>\n<li><strong>Decoder SWA bounded replay:</strong> reduce decoder prefill work by replaying only a bounded suffix. Set <code>DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact</code> and <code>DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140</code> for the default V4.1 settings; the retained suffix must cover the SWA window plus the maximum backtracking length.</li>\n<li><strong>Asynchronous scheduling:</strong> overlap CPU batch scheduling with device execution. Enabled by default.</li>\n<li><strong>Batch-invariant inference:</strong> keep model results identical across different batch compositions. Enable with <code>DEEPINFER_BATCH_INVARIANT=1</code> (NVIDIA only).</li>\n<li><strong>Deterministic sampling:</strong> produce identical outputs for identical input tokens and sampling parameters. Set <code>DEEPINFER_BATCH_INVARIANT=1 DEEPINFER_ENABLE_MTP_VARLEN=0</code> and pass a fixed <code>seq_seed</code> to <code>SamplingParams</code> (NVIDIA only).</li>\n<li><strong>Structured outputs:</strong> constrain generation to a regex, JSON schema, or EBNF grammar. Pass <code>so_args={&quot;schema_type&quot;: ..., &quot;schema&quot;: ...}</code> to <code>SamplingParams</code>; see <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://github.com/deepseek-ai/DeepInfer/blob/main/examples/_structured_output.py\">examples</a>.</li>\n</ul>\n<p>Supported hardware:</p>\n<ul>\n<li>NVIDIA SM100 GPUs</li>\n<li>Huawei Ascend 950 NPUs</li>\n</ul>\n<p>Required for a complete inference service (not included in this release):</p>\n<ul>\n<li>A KV cache system</li>\n<li>VE/Prefill/Decode disaggregation (most required engine functionality is already implemented)</li>\n<li>Load balancing across data-parallel replicas for both prefill and decode</li>\n</ul>\n<h2>Getting started</h2>\n<h3>Prerequisites</h3>\n<ul>\n<li>Linux</li>\n<li>GCC &gt;= 13.0</li>\n<li>Python 3.12</li>\n<li>CUDA 13 (NVIDIA only)</li>\n<li>CANN 9.2 (Ascend only)</li>\n</ul>\n<h3>Build from source</h3>\n<p>Before building DeepInfer, please create and activate an empty Python 3.12 environment.</p>\n<pre><code class=\"language-shell\"><span class=\"code-snippet\" data-language=\"Shell\"><span class=\"syntax-source syntax-shell syntax-bash\"><span class=\"syntax-comment syntax-line syntax-number-sign syntax-shell\"><span class=\"syntax-punctuation syntax-definition syntax-comment syntax-begin syntax-shell\">#</span></span><span class=\"syntax-comment syntax-line syntax-number-sign syntax-shell\"> Use `ascend` instead for the Ascend platform</span><span class=\"syntax-comment syntax-line syntax-number-sign syntax-shell\">\n</span><span class=\"syntax-comment syntax-line syntax-number-sign syntax-shell\"><span class=\"syntax-punctuation syntax-definition syntax-comment syntax-begin syntax-shell\">#</span></span><span class=\"syntax-comment syntax-line syntax-number-sign syntax-shell\"> Keep it exported in the environment where you run DeepInfer</span><span class=\"syntax-comment syntax-line syntax-number-sign syntax-shell\">\n</span><span class=\"syntax-meta syntax-function-call syntax-shell\"><span class=\"syntax-storage syntax-modifier syntax-shell\">export</span> <span class=\"syntax-variable syntax-other syntax-readwrite syntax-assignment syntax-shell\">DEEPINFER_ACCELERATOR</span><span class=\"syntax-keyword syntax-operator syntax-assignment syntax-shell\">=</span><span class=\"syntax-string syntax-unquoted syntax-shell\">nvidia</span></span>\n<span class=\"syntax-meta syntax-function-call syntax-shell\"><span class=\"syntax-variable syntax-function syntax-shell\">./build.sh</span></span>\n</span></span></code></pre>\n<h3>Offline inference</h3>\n<p>The default checkpoint, <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017\"><code>deepseek-ai/DeepSeek-V4.1-Flash-1017</code></a>, is downloaded into the Hugging Face cache on the first run; set <code>DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017</code> to run from a local copy instead.</p>\n<pre><code class=\"language-shell\"><span class=\"code-snippet\" data-language=\"Shell\"><span class=\"syntax-source syntax-shell syntax-bash\"><span class=\"syntax-meta syntax-function-call syntax-shell\"><span class=\"syntax-variable syntax-function syntax-shell\">python</span></span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-shell\"> examples/offline_inference.py</span>\n</span></span></code></pre>\n<p>See <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://github.com/deepseek-ai/DeepInfer/blob/main/examples/README.md\">examples</a> for details.</p>\n<h2>Performance benchmarks</h2>\n<table>\n<thead>\n<tr>\n<th>Prefill</th>\n<th>Decode</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://github.com/deepseek-ai/DeepInfer/blob/main/doc/images/prefill_pareto.png\"><picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://github.com/deepseek-ai/DeepInfer/raw/main/doc/images/prefill_pareto.png\" alt=\"Prefill: non-cached input throughput vs. latency\" loading=\"eager\" decoding=\"async\" fetchpriority=\"high\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture></a></td>\n<td><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://github.com/deepseek-ai/DeepInfer/blob/main/doc/images/decode_pareto.png\"><picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://github.com/deepseek-ai/DeepInfer/raw/main/doc/images/decode_pareto.png\" alt=\"Decode: output throughput vs. per-user speed\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture></a></td>\n</tr>\n</tbody>\n</table>\n<p>Measured on 4 SM100 GPUs for prefill and 64 for decode — each within one scale-up domain. To reproduce the benchmarks, see <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://github.com/deepseek-ai/DeepInfer/blob/main/examples/sweep/run_sweep.sh\">run_sweep.sh</a>.</p>\n<h2>Evaluation benchmarks</h2>\n<p>See <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://github.com/deepseek-ai/DeepInfer/blob/main/eval/README.md\">eval/README.md</a>.</p>\n<h2>License</h2>\n<p>DeepInfer is released under the Apache License 2.0. See <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://github.com/deepseek-ai/DeepInfer/blob/main/LICENSE\">LICENSE</a> for details.</p>\n<h2>Acknowledgments</h2>\n<p>DeepInfer was originally built primarily on <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://github.com/vllm-project/vllm\">vLLM</a>, with additional inspiration from <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://github.com/sgl-project/sglang\">SGLang</a> and <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://github.com/NVIDIA/TensorRT-LLM\">TensorRT-LLM</a>. We thank their developers.</p>\n<h2>Citation</h2>\n<pre><code class=\"language-text-bibtex\"><span class=\"code-snippet\" data-language=\"Text-bibtex\">@misc{deepinfer2026,\n  title = {DeepInfer: High-performance, large-scale inference framework for DeepSeek models},\n  author = {DeepSeek-AI},\n  year = {2026},\n  publisher = {GitHub},\n  howpublished = {\\url{https://github.com/deepseek-ai/DeepInfer}},\n}\n</span></code></pre>\n",
      "date_modified": "2026-10-20T09:08:54+00:00",
      "date_published": "2026-10-20T09:08:54+00:00",
      "external_url": "https://github.com/deepseek-ai/DeepInfer",
      "id": "urn:aggr:item:dfdf82203fa140acdefbe4452bc42c191a1168fc",
      "image": "https://opengraph.githubassets.com/2f1c2fc17da059a0f4782846a6f07a98efb17f74bbfdcd9195516247e979bcfc/deepseek-ai/DeepInfer",
      "summary": "DeepInfer is the high-performance inference framework that powers DeepSeek's large-scale production services. This open-source release provides offline inference with the core features needed for serving at scale. Supported models:…",
      "tags": [
        "news"
      ],
      "title": "DeepSeek just open-sourced its inference engine",
      "url": "https://aggr.aymericbeaumet.com/items/hnrss-org/2026-10-20-deepseek-just-open-sourced-its-inference-engine/"
    }
  ],
  "language": "en",
  "title": "github.com/deepseek-ai",
  "version": "https://jsonfeed.org/version/1.1"
}