{
  "@context": "https://schema.org",
  "@id": "https://aggr.aymericbeaumet.com/items/simonwillison-net/2026-09-17-bonsai-2-27b-near-lossless-compression-in-a-9x-smaller/#webpage",
  "@type": [
    "WebPage",
    "ArchiveComponent"
  ],
  "articleBody": "If you want to try out out the GGUFs from [https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#th...](https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#these-files-need-our-llamacpp-build) be aware that you need Prism's llama.cpp fork to get them to work, from [https://github.com/PrismML-Eng/llama.cpp/releases/tag/prism-...](https://github.com/PrismML-Eng/llama.cpp/releases/tag/prism-b10685-7dffb15)\n\nThis should work:\n\n```\ncd /tmp\n\n# Get the Prism macOS runtime\ncurl -fL https://github.com/PrismML-Eng/llama.cpp/releases/download/prism-b10685-7dffb15/llama-prism-b10685-7dffb15-bin-macos-arm64.tar.gz -o bonsai-runtime.tar.gz\ntar -xzf bonsai-runtime.tar.gz\n\n# Get the ~5.95 GB GGUF model:\ncurl -fL https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf/resolve/main/Ternary-Bonsai-2-27B-PTQ1_0.gguf -o Ternary-Bonsai-2-27B-PTQ1_0.gguf\n\n# Run the server, I used port 8331\n./llama-prism-b10685-7dffb15/llama-server \\\n  -m Ternary-Bonsai-2-27B-PTQ1_0.gguf \\\n  --port 8331 -ngl 99 -fa on -c 32768\n```\n\nThen open http://localhost:8331 for the (very good) baked in llama-server web UI... or run a prompt via the API like this:\n\n```\nuvx llm openai endpoint http://127.0.0.1:8331/v1 \\\n  --model bonsai-2-27b --responses hi\n```\n\nThat's running at ~20 token/second for me on an M5 Pro (after a server restart I got 44 token/second, not sure why), but I'm pretty sure something isn't working right, on startup the server said \"`ggml_metal_device_init: - the tensor API is not supported in this environment - disabling`\".\n",
  "authors": [],
  "captured_at": "2026-09-25T18:30:37.122772579+00:00",
  "category": "programming",
  "content_html": "<p>If you want to try out out the GGUFs from <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#these-files-need-our-llamacpp-build\">https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#th...</a> be aware that you need Prism's llama.cpp fork to get them to work, from <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://github.com/PrismML-Eng/llama.cpp/releases/tag/prism-b10685-7dffb15\">https://github.com/PrismML-Eng/llama.cpp/releases/tag/prism-...</a></p>\n<p>This should work:</p>\n<pre><code><span class=\"code-snippet\" data-language=\"Shell\"><span class=\"syntax-source syntax-shell syntax-bash\"><span class=\"syntax-meta syntax-function-call syntax-shell\"><span class=\"syntax-support syntax-function syntax-cd syntax-shell\">cd</span></span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-shell\"> /tmp</span>\n\n<span class=\"syntax-comment syntax-line syntax-number-sign syntax-shell\"><span class=\"syntax-punctuation syntax-definition syntax-comment syntax-begin syntax-shell\">#</span></span><span class=\"syntax-comment syntax-line syntax-number-sign syntax-shell\"> Get the Prism macOS runtime</span><span class=\"syntax-comment syntax-line syntax-number-sign syntax-shell\">\n</span><span class=\"syntax-meta syntax-function-call syntax-shell\"><span class=\"syntax-variable syntax-function syntax-shell\">curl</span></span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-shell\"><span class=\"syntax-variable syntax-parameter syntax-option syntax-shell\"><span class=\"syntax-punctuation syntax-definition syntax-parameter syntax-shell\"> -</span>fL</span> https://github.com/PrismML-Eng/llama.cpp/releases/download/prism-b10685-7dffb15/llama-prism-b10685-7dffb15-bin-macos-arm64.tar.gz<span class=\"syntax-variable syntax-parameter syntax-option syntax-shell\"><span class=\"syntax-punctuation syntax-definition syntax-parameter syntax-shell\"> -</span>o</span> bonsai-runtime.tar.gz</span>\n<span class=\"syntax-meta syntax-function-call syntax-shell\"><span class=\"syntax-variable syntax-function syntax-shell\">tar</span></span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-shell\"><span class=\"syntax-variable syntax-parameter syntax-option syntax-shell\"><span class=\"syntax-punctuation syntax-definition syntax-parameter syntax-shell\"> -</span>xzf</span> bonsai-runtime.tar.gz</span>\n\n<span class=\"syntax-comment syntax-line syntax-number-sign syntax-shell\"><span class=\"syntax-punctuation syntax-definition syntax-comment syntax-begin syntax-shell\">#</span></span><span class=\"syntax-comment syntax-line syntax-number-sign syntax-shell\"> Get the ~5.95 GB GGUF model:</span><span class=\"syntax-comment syntax-line syntax-number-sign syntax-shell\">\n</span><span class=\"syntax-meta syntax-function-call syntax-shell\"><span class=\"syntax-variable syntax-function syntax-shell\">curl</span></span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-shell\"><span class=\"syntax-variable syntax-parameter syntax-option syntax-shell\"><span class=\"syntax-punctuation syntax-definition syntax-parameter syntax-shell\"> -</span>fL</span> https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf/resolve/main/Ternary-Bonsai-2-27B-PTQ1_0.gguf<span class=\"syntax-variable syntax-parameter syntax-option syntax-shell\"><span class=\"syntax-punctuation syntax-definition syntax-parameter syntax-shell\"> -</span>o</span> Ternary-Bonsai-2-27B-PTQ1_0.gguf</span>\n\n<span class=\"syntax-comment syntax-line syntax-number-sign syntax-shell\"><span class=\"syntax-punctuation syntax-definition syntax-comment syntax-begin syntax-shell\">#</span></span><span class=\"syntax-comment syntax-line syntax-number-sign syntax-shell\"> Run the server, I used port 8331</span><span class=\"syntax-comment syntax-line syntax-number-sign syntax-shell\">\n</span><span class=\"syntax-meta syntax-function-call syntax-shell\"><span class=\"syntax-variable syntax-function syntax-shell\">./llama-prism-b10685-7dffb15/llama-server</span></span><span class=\"syntax-meta syntax-function-call syntax-arguments syntax-shell\"> <span class=\"syntax-punctuation syntax-separator syntax-continuation syntax-line syntax-shell\">\\\n</span><span class=\"syntax-variable syntax-parameter syntax-option syntax-shell\"><span class=\"syntax-punctuation syntax-definition syntax-parameter syntax-shell\">  -</span>m</span> Ternary-Bonsai-2-27B-PTQ1_0.gguf <span class=\"syntax-punctuation syntax-separator syntax-continuation syntax-line syntax-shell\">\\\n</span><span class=\"syntax-variable syntax-parameter syntax-option syntax-shell\"><span class=\"syntax-punctuation syntax-definition syntax-parameter syntax-shell\">  --</span>port</span> 8331<span class=\"syntax-variable syntax-parameter syntax-option syntax-shell\"><span class=\"syntax-punctuation syntax-definition syntax-parameter syntax-shell\"> -</span>ngl</span> 99<span class=\"syntax-variable syntax-parameter syntax-option syntax-shell\"><span class=\"syntax-punctuation syntax-definition syntax-parameter syntax-shell\"> -</span>fa</span> on<span class=\"syntax-variable syntax-parameter syntax-option syntax-shell\"><span class=\"syntax-punctuation syntax-definition syntax-parameter syntax-shell\"> -</span>c</span> 32768</span>\n</span></span></code></pre>\n<p>Then open <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"http://localhost:8331\">http://localhost:8331</a> for the (very good) baked in llama-server web UI... or run a prompt via the API like this:</p>\n<pre><code><span class=\"code-snippet\">uvx llm openai endpoint http://127.0.0.1:8331/v1 \\\n  --model bonsai-2-27b --responses hi\n</span></code></pre>\n<p>That's running at ~20 token/second for me on an M5 Pro (after a server restart I got 44 token/second, not sure why), but I'm pretty sure something isn't working right, on startup the server said &quot;<code>ggml_metal_device_init: - the tensor API is not supported in this environment - disabling</code>&quot;.</p>\n",
  "content_markdown": "If you want to try out out the GGUFs from [https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#th...](https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#these-files-need-our-llamacpp-build) be aware that you need Prism's llama.cpp fork to get them to work, from [https://github.com/PrismML-Eng/llama.cpp/releases/tag/prism-...](https://github.com/PrismML-Eng/llama.cpp/releases/tag/prism-b10685-7dffb15)\n\nThis should work:\n\n```\ncd /tmp\n\n# Get the Prism macOS runtime\ncurl -fL https://github.com/PrismML-Eng/llama.cpp/releases/download/prism-b10685-7dffb15/llama-prism-b10685-7dffb15-bin-macos-arm64.tar.gz -o bonsai-runtime.tar.gz\ntar -xzf bonsai-runtime.tar.gz\n\n# Get the ~5.95 GB GGUF model:\ncurl -fL https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf/resolve/main/Ternary-Bonsai-2-27B-PTQ1_0.gguf -o Ternary-Bonsai-2-27B-PTQ1_0.gguf\n\n# Run the server, I used port 8331\n./llama-prism-b10685-7dffb15/llama-server \\\n  -m Ternary-Bonsai-2-27B-PTQ1_0.gguf \\\n  --port 8331 -ngl 99 -fa on -c 32768\n```\n\nThen open http://localhost:8331 for the (very good) baked in llama-server web UI... or run a prompt via the API like this:\n\n```\nuvx llm openai endpoint http://127.0.0.1:8331/v1 \\\n  --model bonsai-2-27b --responses hi\n```\n\nThat's running at ~20 token/second for me on an M5 Pro (after a server restart I got 44 token/second, not sure why), but I'm pretty sure something isn't working right, on startup the server said \"`ggml_metal_device_init: - the tensor API is not supported in this environment - disabling`\".\n",
  "dateCreated": "2026-09-25T18:30:37.122772579+00:00",
  "date_created": "2026-09-17T22:13:05+00:00",
  "date_modified": "2026-09-17T22:13:05+00:00",
  "date_published": "2026-09-17T22:13:05+00:00",
  "description": "My comment on Bonsai 2 27B: Near-Lossless Compression in a 9x Smaller Footprint — Hacker News. If you want to try out out the GGUFs from https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#th... be aware that you need Prism's…",
  "external_url": "https://simonwillison.net/2026/Sep/17/hn-49747390/",
  "git": {
    "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e",
    "path": "items/simonwillison-net/2026/09/2026-09-17-bonsai-2-27b-near-lossless-compression-in-a-9x-smaller.md",
    "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/simonwillison-net/2026/09/2026-09-17-bonsai-2-27b-near-lossless-compression-in-a-9x-smaller.md"
  },
  "id": "urn:aggr:item:fe4625ff2ebcc6a2f6d5c34dbf60f92b5fa1c690",
  "isBasedOn": {
    "@id": "https://simonwillison.net/2026/Sep/17/hn-49747390/"
  },
  "mainEntity": {
    "@id": "https://simonwillison.net/2026/Sep/17/hn-49747390/",
    "@type": "CreativeWork",
    "archivedAt": {
      "@id": "https://aggr.aymericbeaumet.com/items/simonwillison-net/2026-09-17-bonsai-2-27b-near-lossless-compression-in-a-9x-smaller/#webpage"
    },
    "dateModified": "2026-09-17T22:13:05+00:00",
    "datePublished": "2026-09-17T22:13:05+00:00",
    "headline": "Bonsai 2 27B: Near-Lossless Compression in a 9x Smaller Footprint",
    "timeRequired": "PT1M",
    "url": "https://simonwillison.net/2026/Sep/17/hn-49747390/",
    "wordCount": 225
  },
  "name": "Archived snapshot: Bonsai 2 27B: Near-Lossless Compression in a 9x Smaller Footprint",
  "preview": null,
  "reading_minutes": 1,
  "replicated_at": null,
  "source": "simonwillison-net",
  "source_name": "Simon Willison's Weblog",
  "summary": "My comment on Bonsai 2 27B: Near-Lossless Compression in a 9x Smaller Footprint — Hacker News. If you want to try out out the GGUFs from https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#th... be aware that you need Prism's llama.cpp fork to get them to work, from https://github.com/PrismML-Eng/llama.cpp/releases/tag/prism-... This should work: cd /tmp # Get the Prism macOS runtime curl -fL https://github.com/PrismML-Eng/llama.cpp/releases/download/prism-b10685-7dffb15/llama-prism-b10685-7dffb15-bin-macos-arm64.tar.gz -o bonsai-runtime.tar.gz tar -xzf bonsai-runtime.tar.gz # Get the ~5.95 GB GGUF model: curl -fL https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf/resolve/main/Ternary-Bonsai-2-27B-PTQ1_0.gguf -o Ternary-Bonsai-2-27B-PTQ1_0.gguf # Run the server, I used port 8331 ./llama-prism-b10685-7dffb15/llama-server \\ -m Ternary-Bonsai-2-27B-PTQ1_0.gguf \\ --port 8331 -ngl 99 -fa on -c 32768 Then open http://localhost:8331 for the (very good) baked in llama-server web UI... or run a prompt via the API like this: uvx llm openai endpoint http://127.0.0.1:8331/v1 \\ --model bonsai-2-27b --responses hi That's running at ~20 token/second for me on an M5 Pro (after a server restart I got 44 token/second, not sure why), but I'm pretty sure something isn't working right, on startup the server said \"ggml_metal_device_init: - the tensor API is not supported in this environment - disabling\". Tags: llama-cpp",
  "tags": [
    "llama-cpp"
  ],
  "temporalCoverage": "2026-09-25T18:30:37.122772579+00:00",
  "title": "Bonsai 2 27B: Near-Lossless Compression in a 9x Smaller Footprint",
  "url": "https://aggr.aymericbeaumet.com/items/simonwillison-net/2026-09-17-bonsai-2-27b-near-lossless-compression-in-a-9x-smaller/",
  "word_count": 225
}