llama.cpp server now ships with router mode, which lets you dynamically load, unload, and switch between multiple models without restarting.
\n\n\nReminder: llama.cpp server is a lightweight, OpenAI-compatible HTTP server for running LLMs locally.
\n
This feature was a popular request to bring Ollama-style model management to llama.cpp. It uses a multi-process architecture where each model runs in its own process, so if one model crashes, others remain unaffected.
\nStart the server in router mode by not specifying a model:
\n\nThis auto-discovers models from your llama.cpp cache (LLAMA_CACHE or ~/.cache/llama.cpp). If you've previously downloaded models via llama-server -hf user/model, they'll be available automatically.
You can also point to a local directory of GGUF files:
\n\n--models-dir folder for GGUF files--models-max (default: 4), the least-recently-used model unloadsmodel field in your request determines which model handles it\nOn the first request, the server automatically loads the model into memory (loading time depends on model size). Subsequent requests to the same model are instant since it's already loaded.
\n\nReturns all discovered models with their status (loaded, loading, or unloaded).
\n\n| Flag | \nDescription | \n
|---|---|
--models-dir PATH | \nDirectory containing your GGUF files | \n
--models-max N | \nMax models loaded simultaneously (default: 4) | \n
--no-models-autoload | \nDisable auto-loading; require explicit /models/load calls | \n
All model instances inherit settings from the router:
\n\nAll loaded models will use 8192 context and full GPU offload. You can also define per-model settings using presets:
\n\n[my-model]\nmodel = /path/to/model.gguf\nctx-size = 65536\ntemp = 0.7\nThe built-in web UI also supports model switching. Just select a model from the dropdown and it loads automatically.
\nWe hope this feature makes it easier to A/B test different model versions, run multi-tenant deployments, or simply switch models during development without restarting the server.
\nHave questions or feedback? Drop a comment below or open an issue on GitHub.
\n", "content_markdown": "[llama.cpp server](https://github.com/ggml-org/llama.cpp/tree/master/tools/server) now ships with **router mode**, which lets you dynamically load, unload, and switch between multiple models without restarting.\n\n> Reminder: llama.cpp server is a lightweight, OpenAI-compatible HTTP server for running LLMs locally.\n\nThis feature was a popular request to bring Ollama-style model management to llama.cpp. It uses a multi-process architecture where each model runs in its own process, so if one model crashes, others remain unaffected.\n\n## Quick Start\n\nStart the server in router mode by **not specifying a model**:\n\n```bash\nllama-server\n```\n\nThis auto-discovers models from your llama.cpp cache (`LLAMA_CACHE` or `~/.cache/llama.cpp`). If you've previously downloaded models via `llama-server -hf user/model`, they'll be available automatically.\n\nYou can also point to a local directory of GGUF files:\n\n```bash\nllama-server --models-dir ./my-models\n```\n\n## Features\n\n1. **Auto-discovery**: Scans your llama.cpp cache (default) or a custom `--models-dir` folder for GGUF files\n2. **On-demand loading**: Models load automatically when first requested\n3. **LRU eviction**: When you hit `--models-max` (default: 4), the least-recently-used model unloads\n4. **Request routing**: The `model` field in your request determines which model handles it\n\n## Examples\n\n### Chat with a specific model\n\n```bash\ncurl http://localhost:8080/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"ggml-org/gemma-3-4b-it-GGUF:Q4_K_M\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hello!\"}]\n }'\n```\n\nOn the first request, the server automatically loads the model into memory (loading time depends on model size). Subsequent requests to the same model are instant since it's already loaded.\n\n### List available models\n\n```bash\ncurl http://localhost:8080/models\n```\n\nReturns all discovered models with their status (`loaded`, `loading`, or `unloaded`).\n\n### Manually load a model\n\n```bash\ncurl -X POST http://localhost:8080/models/load \\\n -H \"Content-Type: application/json\" \\\n -d '{\"model\": \"my-model.gguf\"}'\n```\n\n### Unload a model to free VRAM\n\n```bash\ncurl -X POST http://localhost:8080/models/unload \\\n -H \"Content-Type: application/json\" \\\n -d '{\"model\": \"my-model.gguf\"}'\n```\n\n## Key Options\n\n| Flag | Description |\n| ---------------------- | ----------------------------------------------------------- |\n| `--models-dir PATH` | Directory containing your GGUF files |\n| `--models-max N` | Max models loaded simultaneously (default: 4) |\n| `--no-models-autoload` | Disable auto-loading; require explicit `/models/load` calls |\n\nAll model instances inherit settings from the router:\n\n```bash\nllama-server --models-dir ./models -c 8192 -ngl 99\n```\n\nAll loaded models will use 8192 context and full GPU offload. You can also define per-model settings using [presets](https://github.com/ggml-org/llama.cpp/pull/17859):\n\n```bash\nllama-server --models-preset config.ini\n```\n\n```ini\n[my-model]\nmodel = /path/to/model.gguf\nctx-size = 65536\ntemp = 0.7\n```\n\n## Also available in the Web UI\n\nThe [built-in web UI](https://github.com/ggml-org/llama.cpp/tree/master/tools/server/webui) also supports model switching. Just select a model from the dropdown and it loads automatically.\n\n## Join the Conversation\n\nWe hope this feature makes it easier to A/B test different model versions, run multi-tenant deployments, or simply switch models during development without restarting the server.\n\nHave questions or feedback? Drop a comment below or open an issue on [GitHub](https://github.com/ggml-org/llama.cpp/issues).\n", "dateCreated": "2026-10-04T12:50:30.804479567+00:00", "date_created": "2025-12-11T15:47:44+00:00", "date_modified": "2025-12-11T15:47:44+00:00", "date_published": "2025-12-11T15:47:44+00:00", "description": "llama.cpp server now ships with router mode, which lets you dynamically load, unload, and switch between multiple models without restarting. Reminder: llama.cpp server is a lightweight, OpenAI-compatible HTTP server for running LLMs…", "external_url": "https://huggingface.co/blog/ggml-org/model-management-in-llamacpp", "git": { "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e", "path": "items/huggingface-co/2025/12/2025-12-11-new-in-llama-cpp-model-management.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/huggingface-co/2025/12/2025-12-11-new-in-llama-cpp-model-management.md" }, "id": "urn:aggr:item:675a3aa73fc26d1164b2f82a9c5c3eb8be1f4515", "isBasedOn": { "@id": "https://huggingface.co/blog/ggml-org/model-management-in-llamacpp" }, "mainEntity": { "@id": "https://huggingface.co/blog/ggml-org/model-management-in-llamacpp", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2025-12-11-new-in-llama-cpp-model-management/#webpage" }, "dateModified": "2025-12-11T15:47:44+00:00", "datePublished": "2025-12-11T15:47:44+00:00", "headline": "New in llama.cpp: Model Management", "timeRequired": "PT3M", "url": "https://huggingface.co/blog/ggml-org/model-management-in-llamacpp", "wordCount": 464 }, "name": "Archived snapshot: New in llama.cpp: Model Management", "preview": null, "reading_minutes": 3, "replicated_at": null, "source": "huggingface-co", "source_name": "Hugging Face - Blog", "summary": null, "tags": [], "temporalCoverage": "2026-10-04T12:50:30.804479567+00:00", "title": "New in llama.cpp: Model Management", "url": "https://aggr.aymericbeaumet.com/items/huggingface-co/2025-12-11-new-in-llama-cpp-model-management/", "word_count": 464 }