{ "@context": "https://schema.org", "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-08-10-meta-is-back-with-muse-glimmer-local-agentic-multimodal-and/#webpage", "@type": [ "WebPage", "ArchiveComponent" ], "articleBody": "Great news from the OGs of open source LLMs! Muse Glimmer, released today, is Meta’s new multimodal model, especially designed for local agentic use cases. Distilled from Muse to **30B** parameters, and released under the **Apache 2.0 license**, it’s ideal deploying locally for privacy, reducing costs, or just hacking around. It’s intended for privacy-aware applications such as coding, document analysis, personal assistants, Claw- or Hermes-like setups.\n\nTo celebrate, we are shipping with Meta day-0 support in `transformers`, `llama.cpp`, `vLLM`, Inference Endpoints, and other libraries. We built a few cool things and explain our findings in this blog.\n\n[**Check out the demos below for inspiration.**](https://huggingface.co/blog/muse-glimmer#demos)\n\nYou can find [Muse Glimmer on the Hugging Face Hub](https://huggingface.co/meta-models/Muse-Glimmer-30B).\n\n## Benchmarks\n\nBenchmark results\n\nScores are reported as published. **Bold** indicates the best result among the compared models; ↓ indicates lower is better.\n\n| Category | Benchmark | Muse Glimmer-30B High Reasoning | Gemma4-31B Thinking Mode | Qwen3.6-27B Thinking Mode |\n| ---------------------------------- | -------------------------------------- | ------------------------------------------- | ----------------------------------------------- | ------------------------------------------- |\n| General Agentic | MCP Atlas | **75.5** | 54.2 | 62.5 |\n| General Agentic | DeepSearch QA | **74.6** | 61.7 | 71.1 |\n| General Agentic | τ³-Banking | **23.5** | 15.1 | 16.7 |\n| General Agentic | WildClawBench | **47.6** | 37.6 | 43.2 |\n| General Agentic | GDPval-AA | 953 | 811 | **1141** |\n| General Agentic | GAIA2 | **43.3** | 36.4 | 40.0 |\n| General Agentic | SkillsBench (With Skills) | 44.3 | 32.4 | **46.6** |\n| General Agentic | OSWorld-Verified | 65.9 | 58.5 | **75.6** |\n| Agentic Coding | SWE-Bench Pro | **51.2** | 36.9 | 50.2 |\n| Agentic Coding | SWE-Bench Verified | 76.0 | 66.6 | **77.2** |\n| Agentic Coding | TerminalBench 2.1 | 51.7 | 43.4 | **60.7** |\n| Agentic Coding | SciCode | **43.6** | 43.4 | 39.8 |\n| Multimodal | Charxiv Reasoning | **78.8** | 77.7 | 78.4 |\n| Multimodal | ScreenSpot Pro | 75.4 | 75.9 | **76.1** |\n| Multimodal | OmniDocBench v1.5 | 75.8 | 72.5 | **77.8** |\n| Multimodal | MMMU Pro | 74 | 73 | **75** |\n| Safety | CI Memories | Violation (↓): 26.4 Coverage: 64.8 | **Violation (↓): 12.1** Coverage: 53.0 | Violation (↓): 53.4 Coverage: 66.9 |\n| Safety | Siren AgentDojo | Attack Success Rate (↓): 28.4 Utility: 94.2 | **Attack Success Rate (↓): 25.6** Utility: 90.8 | Attack Success Rate (↓): 40.3 Utility: 92.7 |\n| General Capabilities and Reasoning | IFBench | **77.0** | 76.0 | 70.8 |\n| General Capabilities and Reasoning | AIME 2026 | **94.7** | 89.2 | 94.1 |\n| General Capabilities and Reasoning | GPQA Diamond | 83.5 | **85.7** | 84.2 |\n| General Capabilities and Reasoning | Humanity’s Last Exam (Text + No Tools) | 22.0 | **23.6** | 23.1 |\n| General Capabilities and Reasoning | AA-LCR | **80.0** | 68.3 | 73.3 |\n| General Capabilities and Reasoning | Beam 128K | **65.1** | 58.2 | 63.0 |\n\n## Architecture\n\nMuse Glimmer is a dense 30B parameter model consisting of:\n\n- 2B ViT-style encoder for vision (Perception Encoder)\n- 28B parameter text decoder\n\nIn addition to the main VLM, there’s also a speculative decoding drafter implemented on DFlash. Usage of this module is optional, and it can provide much faster generation in exchange for some memory cost. We found this drafter to be particularly well suited to structured content generation such as coding.\n\n### Text Decoder\n\nThe language model uses the following architecture components:\n\n- **Hybrid attention:** Alternating between three sliding window layers (of 2,048 tokens) using rotary position embedding, followed by a fourth layer that uses full attention and NoPE (no positional embedding). The pattern is therefore (SWA, SWA, SWA, Full), repeated 13 times to a total of 52 layers. This allows the model to retain relative order and distance information with RoPE and preserve information globally with NoPE.\n- **Gated Grouped-Query Attention:** Each key-value head is shared by 16 query heads, which reduces KV-cache memory by 16x and makes generation faster and cheaper.\n- **Q-K normalization with extra query scaling:** Before computing attention, Muse Glimmer applies RMS normalization to every query and key head to keep attention logits stable. After this, queries are multiplied by a scale factor to set the target logit scale after normalization. The extra query scaling behaves like an inverse temperature at the softmax level.\n\n### Perception Encoder\n\nMuse Glimmer uses one image encoder to handle both images and videos. Unlike the relatively small vision encoders used in other VLMs, this is a sizable 2B ViT-like model designed after the Perception Encoder architecture. Perception Encoder was previously introduced by Meta [as a backbone for various downstream spatial and multimodal tasks](https://huggingface.co/papers/2504.13181).\\\nThe encoder patchifies images to a shape of 2 frames x 3 channels x 14 x 14, and passes them through a linear layer for projection. An interpolated absolute position embedding from a learned position table is then added to these embeddings. These are then sent to the vision tower which consist of 50 layers and GELU MLPs. Similar to the language model, the attention pattern consists of three window attention layers followed by one full attention layer. Inside the attention layers, 2D RoPE is applied to the queries and keys.\n\nAfter transformer, pixel shuffle concatenates 2x2 groups of neighboring spatial tokens which reduces the number of image tokens 4x without discarding their channels. The merged features are then projected to the shared embedding space of the text decoder.\n\nVideos go through the same encoder frame by frame, where each frame is converted into patches (of shape \\[batch, temporal groups, grid height, grid width, 2 frames, 3 channels, 14, 14\\]). The processor targets 2 frames per second and caps the clip at 96 frames sampled evenly across video. The processor creates timestamped video placeholders, interleaving text with frame e.g. “Time: 0.0s <|video|> x N” in which the final video embeddings are replaced before the final projection layer.\n\n## Transformers\n\nUpgrade transformers to the latest version to be able to use Muse Glimmer.\n\n```bash\npip install --upgrade transformers accelerate\n```\n\nMuse Glimmer comes with day-0 support in transformers, both for the main model and the speculative decoding drafter. You can use `AutoModelForMultimodalLM` and `AutoProcessor` classes to load the model and the processor.\n\n```py\nfrom transformers import AutoProcessor, AutoModelForMultimodalLM\n\nMODEL_ID = \"meta-models/Muse-Glimmer-30B\"\n\n# Load model\nprocessor = AutoProcessor.from_pretrained(MODEL_ID)\nmodel = AutoModelForMultimodalLM.from_pretrained(\n MODEL_ID,\n dtype=\"auto\",\n device_map=\"auto\"\n)\n```\n\nThe same snippet runs unchanged on NVIDIA (CUDA), AMD (ROCm) and Intel (XPU) GPUs, `device_map=\"auto\"` places the model on whichever accelerator is available.\n\n### Text-only Inference\n\nAfter loading the model, you can do text-only inference with it as follows.\n\n```py\nfrom transformers import AutoProcessor, AutoModelForMultimodalLM\n\nMODEL_ID = \"meta-models/Muse-Glimmer-30B\"\n\n# Load model\nprocessor = AutoProcessor.from_pretrained(MODEL_ID)\nmodel = AutoModelForMultimodalLM.from_pretrained(\n MODEL_ID,\n dtype=\"auto\",\n device_map=\"auto\"\n)\n\n# Prompt\nmessages = [\n {\"role\": \"user\", \"content\": \"Write a short joke about saving RAM.\"},\n]\n\n# Process input\ninputs = processor.apply_chat_template(\n messages,\n tokenize=True,\n return_dict=True,\n return_tensors=\"pt\",\n add_generation_prompt=True,\n reasoning_strength=\"low\"\n).to(model.device)\ninput_len = inputs[\"input_ids\"].shape[-1]\n\n# Generate output\noutputs = model.generate(**inputs)\nresponse = processor.decode(outputs[0][input_len:], skip_special_tokens=False)\nprint(response)\n```\n\n### Prompting the model with images and text\n\nWe would need `torchvision` to be able to use images and text.\n\n```bash\npip install torchvision\n```\n\nMuse Glimmer accepts images as input, as demonstrated here:\n\n```py\nfrom transformers import AutoProcessor, AutoModelForMultimodalLM\n\nMODEL_ID = \"meta-models/Muse-Glimmer-30B\"\n\n# Load model\nprocessor = AutoProcessor.from_pretrained(MODEL_ID)\nmodel = AutoModelForMultimodalLM.from_pretrained(\n MODEL_ID,\n dtype=\"auto\",\n device_map=\"auto\"\n)\n\n# Images + Text\nmessages = [\n {\n \"role\": \"user\", \"content\": [\n {\"type\": \"image\", \"image\": \"https://huggingface.co/datasets/merve/vl-test-suite/resolve/main/SF.png\"},\n {\"type\": \"text\", \"text\": \"What is shown in this image?\"}\n ]\n }\n]\n\ninputs = processor.apply_chat_template(\n messages,\n tokenize=True,\n return_dict=True,\n return_tensors=\"pt\",\n add_generation_prompt=True,\n reasoning_strength=\"low\"\n).to(model.device)\ninput_len = inputs[\"input_ids\"].shape[-1]\n\n# Generate output\noutputs = model.generate(**inputs)\nresponse = processor.decode(outputs[0][input_len:], skip_special_tokens=False)\nprint(response)\n```\n\n### Video Inference\n\nTo work with videos we recommend installing `torchcodec` into the environment.\n\n```bash\npip install torchcodec\n```\n\nMuse Glimmer can answer complex questions about videos *without audio*. You can do video inference as follows, here’s an example from VideoMME2, which is the most popular video question answering benchmark.\n\n```py\nfrom transformers import AutoProcessor, AutoModelForMultimodalLM\n\nMODEL_ID = \"meta-models/Muse-Glimmer-30B\"\n\n# Load model\nprocessor = AutoProcessor.from_pretrained(MODEL_ID)\nmodel = AutoModelForMultimodalLM.from_pretrained(\n MODEL_ID,\n dtype=\"auto\",\n device_map=\"auto\"\n)\n\n# Videos + Text\nmessages = [\n {\n \"role\": \"user\",\n \"content\": [\n {\"type\": \"video\", \"video\": \"https://huggingface.co/datasets/merve/vl-test-suite/resolve/main/IMG_8137.mp4\"},\n {\"type\": \"text\", \"text\": \"Describe what happens in this video.\"},\n ],\n },\n]\n\ninputs = processor.apply_chat_template(\n messages,\n tokenize=True,\n return_dict=True,\n return_tensors=\"pt\",\n add_generation_prompt=True,\n reasoning_strength=\"low\",\n processor_kwargs={\"num_frames\": 96},\n).to(model.device)\n\ninput_len = inputs[\"input_ids\"].shape[-1]\noutputs = model.generate(**inputs)\n\nresponse = processor.decode(\n outputs[0, input_len:],\n skip_special_tokens=False,\n)\nprint(response)\n```\n\n### Multimodal tool calling\n\nMuse Glimmer can do multimodal tool calling, here’s how you can do it. In the example below, we ask the model to call the weather tool based on the city in the image.\n\n```py\nimport json\nimport re\nfrom transformers import AutoProcessor, AutoModelForMultimodalLM\n\nMODEL_ID = \"meta-models/Muse-Glimmer-30B\"\n\n# Load model\nprocessor = AutoProcessor.from_pretrained(MODEL_ID)\nmodel = AutoModelForMultimodalLM.from_pretrained(\n MODEL_ID,\n dtype=\"auto\",\n device_map=\"auto\"\n)\n\ntools = [\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"weather.get\",\n \"description\": \"Get the current weather for a city.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"city\": {\"type\": \"string\"},\n },\n \"required\": [\"city\"],\n },\n },\n }\n]\n\n\nmessages = [\n {\n \"role\": \"user\",\n \"content\": [\n {\"type\": \"image\", \"image\": \"https://huggingface.co/datasets/merve/vl-test-suite/resolve/main/SF.png\"},\n {\"type\": \"text\", \"text\": \"I'm going to the city in this picture. What clothes should I wear?\"},\n ],\n },\n]\n\ninputs = processor.apply_chat_template(\n messages,\n tools=tools,\n tokenize=True,\n return_dict=True,\n return_tensors=\"pt\",\n add_generation_prompt=True,\n reasoning_strength=\"low\"\n).to(model.device)\n\ninput_len = inputs[\"input_ids\"].shape[-1]\noutputs = model.generate(**inputs)\nresponse = processor.decode(outputs[0][input_len:], skip_special_tokens=False)\nprint(response)\n```\n\n### Object Detection\n\nYou can use Muse Glimmer to do open ended object detection in images as follows.\n\n```py\nfrom transformers import AutoProcessor, AutoModelForMultimodalLM\n\nMODEL_ID = \"meta-models/Muse-Glimmer-30B\"\n\n# Load model\nprocessor = AutoProcessor.from_pretrained(MODEL_ID)\nmodel = AutoModelForMultimodalLM.from_pretrained(\n MODEL_ID,\n dtype=\"auto\",\n device_map=\"auto\"\n)\n\nmessages = [{\n \"role\": \"user\",\n \"content\": [\n {\"type\": \"image\", \"image\": \"https://huggingface.co/datasets/merve/vl-test-suite/resolve/main/SF.png\"},\n {\n \"type\": \"text\",\n \"text\": (\n \"Detect the bridge. Return only the detection in the model's \"\n \"native object-detection format, with no explanation.\"\n ),\n },\n ],\n}]\n\ninputs = processor.apply_chat_template(\n messages,\n tokenize=True,\n return_dict=True,\n return_tensors=\"pt\",\n add_generation_prompt=True,\n reasoning_strength=\"low\",\n).to(model.device)\n\ninput_len = inputs[\"input_ids\"].shape[-1]\noutputs = model.generate(**inputs, max_new_tokens=128)\nresponse = processor.decode(outputs[0][input_len:], skip_special_tokens=False)\n\ndetections = json.loads(response.removesuffix(\"<|eot|>\"))\nprint(detections)\n```\n\n> Here is an end to end script to perform object detection [GitHub Gist](https://gist.github.com/ariG23498/4f3587eb7753c0ff77c269e2c1efe2c0)\n\n## Llama.cpp\n\nMuse Glimmer comes with day-0 llama.cpp support. Meta has distributed calibrated quants in [this repo](https://huggingface.co/meta-models/Muse-Glimmer-30B-GGUF), and Unsloth is releasing optimized quants as well. DFlash speculative decoding is supported as well. You can use a pre-built llama binary to start a llama server or a CLI. To install llama.cpp, run:\n\n```bash\ncurl -LsSf https://llama.app/install.sh | sh\n```\n\nThen you can start the server as follows.\n\n```bash\nllama serve -hf meta-models/Muse-Glimmer-30B-GGUF\n```\n\nOnce the server has started, you can head to localhost:8080 to chat with the built-in WebUI.\n\nYou can also query the server as follows.\n\n```bash\ncurl http://localhost:8080/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"messages\": [\n {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n {\"role\": \"user\", \"content\": \"Write a limerick about python exceptions\"}\n ]\n }'\n```\n\nYou can also use llama server with coding agents like Pi.\n\n## Speculative Decoding\n\nDFlash uses a lightweight block-diffusion drafter model to provide same output with extra speed-ups in decoding phase. Transformers and llama.cpp ship support for DFlash drafter of Muse Glimmer day-0.\n\nBelow you can see how speculative decoding can speed-up generation in realistic setups. The video shows llama.cpp webui with DFlash on the left and regular generation on the right.\n\n### Speculative Decoding with transformers\n\nYou can load the drafter and model as follows, and infer like how you would with base model with an additional parameter (shown in the upcoming snippets).\n\n```py\nimport torch\nfrom transformers import AutoProcessor, MuseGlimmerAssistantModel, MuseGlimmerForConditionalGeneration\n\nmodel_id = \"meta-models/Muse-Glimmer-30B\"\nassistant_model_id = \"meta-models/Muse-Glimmer-30B-assistant\"\ntarget = MuseGlimmerForConditionalGeneration.from_pretrained(model_id, dtype=torch.bfloat16, device_map=\"auto\")\nassistant = MuseGlimmerAssistantModel.from_pretrained(assistant_model_id, dtype=torch.bfloat16, device_map=\"auto\")\nprocessor = AutoProcessor.from_pretrained(model_id)\n\nmessages = [\n {\n \"role\": \"user\", \"content\": [\n {\"type\": \"image\", \"url\": \"https://huggingface.co/datasets/merve/vl-test-suite/resolve/main/SF.png\"},\n {\"type\": \"text\", \"text\": \"What is shown in this image?\"}\n ]\n }\n]\n\ninputs = processor.apply_chat_template(\n messages,\n tokenize=True,\n return_dict=True,\n return_tensors=\"pt\",\n add_generation_prompt=True,\n reasoning_strength=\"low\"\n).to(target.device)\ninput_len = inputs[\"input_ids\"].shape[-1]\n\noutputs = target.generate(\n **inputs,\n assistant_model=assistant,\n speculation_type=\"dflash\",\n do_sample=True\n)\nresponse = processor.decode(outputs[0][input_len:], skip_special_tokens=False)\nprint(response)\n```\n\n### Speculative Decoding with llama.cpp\n\nYou can start llama server using following command. `--spec-draft-n-max` argument controls how many future tokens DFlash proposes during each speculative-decoding step. Muse Glimmer’s DFlash model was trained with a block size of 16, one anchor token plus 15 proposed tokens, so any value above 15 will be clamped to 15.\n\n```bash\nllama serve -hf meta-models/Muse-Glimmer-30B-GGUF --spec-type draft-dflash --spec-draft-n-max 15\n```\n\nYou can also use llama cli with speculative decoding drafter as follows.\n\n```bash\nllama cli -hf meta-models/Muse-Glimmer-30B-GGUF --spec-type draft-dflash\n```\n\n### Inference Endpoints\n\nFor a managed, autoscaling deployment, open the [Muse Glimmer 30B Inference Endpoints preset](https://endpoints.huggingface.co/huggingface/new/meta-models/Muse-Glimmer-30B). The model is already selected: choose the organization, cloud provider, region, compatible GPU instance, authentication, and autoscaling settings, then review the hourly price and click **Create Endpoint**. Once its status is **Running**, you can test it in the Playground and copy the endpoint URL and model name from the Overview.\n\n[![Deploy Muse Glimmer 30B with Hugging Face Inference Endpoints](https://endpoints.huggingface.co/social-share/thumbnail-main.jpg)](https://endpoints.huggingface.co/huggingface/new/meta-models/Muse-Glimmer-30B)\n\nThe deployed model exposes an OpenAI-compatible Chat Completions API. Keep your Hugging Face token in an environment variable, set `HF_ENDPOINT_URL` to the URL shown in the Overview without `/v1`, and set `HF_ENDPOINT_MODEL` to the endpoint's model name.\n\n```bash\nexport HF_TOKEN=\"hf_...\"\nexport HF_ENDPOINT_URL=\"https://...endpoints.huggingface.cloud\"\nexport HF_ENDPOINT_MODEL=\"\"\npip install --upgrade openai\n```\n\n```python\nimport os\nfrom openai import OpenAI\n\nclient = OpenAI(\n base_url=f\"{os.environ['HF_ENDPOINT_URL'].rstrip('/')}/v1/\",\n api_key=os.environ[\"HF_TOKEN\"],\n)\n\nresponse = client.chat.completions.create(\n model=os.environ[\"HF_ENDPOINT_MODEL\"],\n messages=[\n {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n {\"role\": \"user\", \"content\": \"Write a limerick about Python exceptions.\"},\n ],\n max_tokens=256,\n)\nprint(response.choices[0].message.content)\n```\n\nSee the [Inference Endpoints documentation](https://huggingface.co/docs/inference-endpoints/en/index) for configuration, autoscaling, security, logs, and monitoring.\n\n## Support for Muse Glimmer vLLM with transformers backend\n\nFor this release, we ship support for vLLM with transformers backend.\n\n```bash\n\n# tensor parallel serving across 4 GPUs\nvllm serve meta-models/Muse-Glimmer-30B --model-impl transformers --tensor-parallel-size 4\n\n# infer\ncurl -s http://127.0.0.1:8000/v1/chat/completions \\\n -H 'Content-Type: application/json' \\\n -d '{\n \"model\": \"meta-models/Muse-Glimmer-30B\",\n \"messages\": [\n {\"role\": \"user\", \"content\": \"Explain tensor parallelism briefly.\"}\n ],\n \"temperature\": 0.0,\n \"max_tokens\": 256\n }'\n```\n\n## Fine-tuning with TRL\n\nYou can use TRL to fine-tune Muse Glimmer using various methods from SFT to Async GRPO. We have run two experiments on bf16 with Hopper-class GPUs with 80GB VRAM each.\n\n| Workload | Practical minimum |\n| --------------------------------------- | ------------------------------------------ |\n| Inference / eval, BF16 | 1×80 GB H100 |\n| LoRA SFT, BF16 | 1×80 GB H100, microbatch 1 + checkpointing |\n| Full SFT, BF16 | 8×80 GB H100 with FSDP/ZeRO-3 |\n| LoRA GRPO, Transformers rollouts | 1×80 GB H100, but slow/tight |\n| LoRA GRPO, separate vLLM rollout server | 8×H100: 4 rollout + 4 training |\n| Full-finetune GRPO | 8 GPUs is usually insufficient |\n\nAs part of this release, we ship an example to fine-tune [Muse Glimmer on small split of MolmoWeb dataset](https://huggingface.co/merve/smol-vision/blob/main/qlora_click_grounding.ipynb). This shows how to make model generate structured outputs and how to fine-tune on images.\n\nWe also experimented with running the model on [OpenCode with AsyncGRPO example](https://github.com/huggingface/trl/blob/v1.14.1/examples/async_grpo_opencode/async_grpo_opencode.py). Model shows strong coding capabilities, so we encourage you to try training with coding environments in OpenEnv and TRL.\n\n## Demos\n\nHere are some fun ways to try out Muse Glimmer. In our opinion, the coolest thing about this model is that it is a local scale personal assistant that can code. That means you can make it do things like, quantize itself, find quantized weights on the Hub, deploy itself to inference endpoints, and even optimize itself for specific hardware! Let’s go team local 🚀\n\n## Connect OpenClaw to Muse Glimmer\n\nAssume the Inference Endpoint exposes an OpenAI-compatible `/v1` API.\n\nSet `HF_TOKEN` in the OpenClaw gateway environment, then add this to `~/.openclaw/openclaw.json`:\n\nOpenClaw configuration\n\n```json5\n{\n models: {\n mode: \"merge\",\n providers: {\n muse: {\n baseUrl: \"https://YOUR-ENDPOINT.endpoints.huggingface.cloud/v1\",\n apiKey: {\n source: \"env\",\n provider: \"default\",\n id: \"HF_TOKEN\"\n },\n api: \"openai-completions\",\n authHeader: true,\n models: [{\n id: \"meta-models/Muse-Glimmer-30B\",\n name: \"Muse Glimmer\",\n reasoning: false,\n input: [\"text\", \"image\"],\n contextWindow: 32768,\n maxTokens: 8192\n }]\n }\n }\n },\n agents: {\n defaults: {\n model: { primary: \"muse/meta-models/Muse-Glimmer-30B\" }\n }\n }\n}\n```\n\nRestart OpenClaw:\n\n```bash\nopenclaw gateway restart\n```\n\nValidate from a fresh session:\n\n```bash\nopenclaw agent --message \"Reply with: muse-ready\"\n```\n\nUse the exact model ID returned by the endpoint’s `/v1/models` response if it differs.\n\n### Hey Muse Glimmer, quantize yourself\n\nIf we hook up Muse Glimmer to the [Hugging Face MCP](https://huggingface.co/mcp) and update its [`AGENTS.md`](http://AGENTS.md) we give it the capability to find a quantized version of itself on the hub and run locally. This is handy if you want to work on something private, or just cut costs.\n\nIf you do this a second time, Muse Glimmer will find the cached weights and switch to them, so feel free to add a convenient command like `/spawn`.\n\nMuse Glimmer inspects the machine and Hub, selects or creates a Q4\\_K\\_M GGUF, launches llama-server, and validates model discovery and chat completion. The result is a smaller local build behind an OpenAI-compatible API. Here’s the prompt we added to `AGENTS.md`.\n\n[https://huggingface.co/buckets/huggingface/muse-glimmer-assets/resolve/Muse%20Glimmer%20Quantisation%20Demo%20-%20explained.mp4?download=true](https://huggingface.co/buckets/huggingface/muse-glimmer-assets/resolve/Muse%20Glimmer%20Quantisation%20Demo%20-%20explained.mp4?download=true)\n\nBy adding this to [`AGENTS.md`](http://AGENTS.md) openclaw or hermes will be able to solve the rest.\n\nLocal quantization prompt\n\n```text\n## Local model deployment\n\nWhen asked to deploy locally, perform the work; do not give instructions.\n\n1. Inspect hardware and the Hugging Face cache.\n2. Search the Hub for compatible GGUF weights using `apps=llama.cpp`; confirm exact filenames through the model-tree API.\n3. Prefer an existing suitable GGUF, normally `Q4_K_M`. Treat `mmproj-*.gguf` as projector weights.\n4. If no GGUF exists, download the source weights, convert with `convert_hf_to_gguf.py`, then quantize with `llama-quantize`.\n5. Preserve source weights and record the repository, revision, filenames, and quantization.\n6. Start `llama-server` with an `onyx` alias and an OpenAI-compatible endpoint.\n7. Validate `/v1/models` and `/v1/chat/completions`, requiring non-empty, correct content.\n8. Report concise progress and logs. Claim completion only after validation passes.\n```\n\n### Hey Muse Glimmer, deploy yourself\n\nMuse Glimmer can also take care of the opposite. Let’s get Glimmer to deploy itself on Hugging Face Inference Endpoints. Which is useful if you want to speed up on some cutting edge hardware.\n\nN.B. You can also just deploy [Muse Glimmer to Inference Endpoints](https://endpoints.huggingface.co/huggingface/new/meta-models/Muse-Glimmer-30B) directly and connect your agent.\n\nMuse Glimmer pins the model revision, deploys it to a protected Hugging Face Inference Endpoint, and verifies health, model discovery, and chat completion. It then connects the Claw agent with secrets and rollback preserved. Here’s the prompt we added to [`AGENTS.md`](http://AGENTS.md). Muse glimmer will also need the [Hugging Face MCP](https://huggingface.co/mcp) and/or the [Hugging Face CLI and Skills](https://huggingface.co/docs/hub/en/agents-skills).\n\nInference Endpoint deployment prompt\n\n```text\n## Hugging Face Inference Endpoint deployment\n\nWhen asked to deploy on Hugging Face Inference Endpoints, perform the work; do\nnot give instructions.\n\n1. Inspect Hugging Face authentication, the current model repository, and any\n existing endpoints.\n2. Confirm the exact model repository and immutable revision through the Hub\n API; inspect its architecture, configuration, and chat template.\n3. Confirm that the model is supported by vLLM, then deploy or update a\n protected Inference Endpoint using the managed native vLLM engine.\n4. Choose an available region and the smallest suitable accelerator. Use one\n replica and enable scale-to-zero when supported.\n5. Preserve the previous endpoint configuration for rollback. Do not expose\n tokens, publish private weights, or replace an unrelated endpoint.\n6. Wait for the endpoint to become ready. If startup fails, inspect the logs\n and report the actual blocker rather than repeatedly changing settings.\n7. Validate `/health`, `/v1/models`, and `/v1/chat/completions`, requiring the\n expected model and non-empty, correct content. When agent use is required,\n also validate a real structured tool call.\n8. Configure the Claw agent to use the endpoint's OpenAI-compatible `/v1` URL,\n storing credentials as secrets and retaining the previous provider as\n rollback. Test the connection in a fresh session.\n9. Report concise progress and finish with the repository, revision, engine,\n hardware, endpoint URL, scaling state, and validation results. Claim\n completion only after every required check passes.\n```\n\n### Hey Muse Glimmer, optimize yourself\n\nFinally, let’s get Muse Glimmer to do some light RSI. We can instruct our agent to optimize its own inference engine for specific hardware, in this case a Nvidia H100. To do this, the agent will need to use another inference engine, like Inference Endpoints above.\n\nMuse Glimmer benchmarks its own single-H100 serving stack, testing one reversible change at a time while holding the workload fixed. It keeps only correctness-passing gains and finishes with the fastest reproducible configuration. Here’s the prompt we added to [`AGENTS.md`](http://AGENTS.md). Muse glimmer need the [Hugging Face MCP](https://huggingface.co/mcp) and the [Hugging Face CLI and Skills](https://huggingface.co/docs/hub/en/agents-skills).\n\nSelf-optimization prompt\n\n```text\nYou are Muse Glimmer acting as an autonomous inference-optimization engineer for your own serving stack.\n\nGoal: maximize valid single-H100 aggregate completion throughput in tokens/second.\n\nProtocol:\n1. Establish a correctness-passing baseline.\n2. Test one reversible optimization at a time.\n3. Keep the prompt, concurrency, sampling, request count, warm-up, and decode length fixed.\n4. Reject results that fail correctness or prefix checks.\n5. Record every experiment chronologically with its configuration, raw throughput, correctness, and delta.\n6. Keep improvements and revert regressions.\n7. Stop after six consecutive regressions or when the experiment budget is exhausted.\n8. Report the best valid configuration and exact reproduction command.\n\nCreate a minimal scientific animation of the results:\n- white background;\n- raw tokens/second—never normalize;\n- one point revealed per experiment;\n- connect every point chronologically;\n- begin with the lowest valid result;\n- stop at the best result;\n- export as a GIF.\n\nNever fabricate, interpolate, or count correctness-failing measurements.\n```\n\n[https://huggingface.co/buckets/huggingface/muse-glimmer-assets/resolve/onyx-optimization-progress.gif?download=true](https://huggingface.co/buckets/huggingface/muse-glimmer-assets/resolve/onyx-optimization-progress.gif?download=true)\n\n### Hey Muse Glimmer, research the Hub\n\nTry Muse Glimmer as a Hugging Face research agent. The Gradio Space sends each model request to a private Hugging Face Inference Endpoint through its OpenAI-compatible API. It also connects to the official Hugging Face MCP server, giving the agent read-only tools to search and inspect Hub repositories, models, datasets, Spaces, documentation, and papers.\n\n## Wrapping Up\n\nWe are happy to welcome Muse Glimmer to the Hugging Face Hub. Try [Muse Glimmer](https://huggingface.co/meta-models/Muse-Glimmer-30B) with your local coding setups today!\n", "authors": [], "captured_at": "2026-10-19T01:26:53.998049939+00:00", "category": "ai", "content_html": "

Great news from the OGs of open source LLMs! Muse Glimmer, released today, is Meta’s new multimodal model, especially designed for local agentic use cases. Distilled from Muse to 30B parameters, and released under the Apache 2.0 license, it’s ideal deploying locally for privacy, reducing costs, or just hacking around. It’s intended for privacy-aware applications such as coding, document analysis, personal assistants, Claw- or Hermes-like setups.

\n

To celebrate, we are shipping with Meta day-0 support in transformers, llama.cpp, vLLM, Inference Endpoints, and other libraries. We built a few cool things and explain our findings in this blog.

\n

Check out the demos below for inspiration.

\n

You can find Muse Glimmer on the Hugging Face Hub.

\n

Benchmarks

\n

Benchmark results

\n

Scores are reported as published. Bold indicates the best result among the compared models; ↓ indicates lower is better.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
CategoryBenchmarkMuse Glimmer-30B High ReasoningGemma4-31B Thinking ModeQwen3.6-27B Thinking Mode
General AgenticMCP Atlas75.554.262.5
General AgenticDeepSearch QA74.661.771.1
General Agenticτ³-Banking23.515.116.7
General AgenticWildClawBench47.637.643.2
General AgenticGDPval-AA9538111141
General AgenticGAIA243.336.440.0
General AgenticSkillsBench (With Skills)44.332.446.6
General AgenticOSWorld-Verified65.958.575.6
Agentic CodingSWE-Bench Pro51.236.950.2
Agentic CodingSWE-Bench Verified76.066.677.2
Agentic CodingTerminalBench 2.151.743.460.7
Agentic CodingSciCode43.643.439.8
MultimodalCharxiv Reasoning78.877.778.4
MultimodalScreenSpot Pro75.475.976.1
MultimodalOmniDocBench v1.575.872.577.8
MultimodalMMMU Pro747375
SafetyCI MemoriesViolation (↓): 26.4 Coverage: 64.8Violation (↓): 12.1 Coverage: 53.0Violation (↓): 53.4 Coverage: 66.9
SafetySiren AgentDojoAttack Success Rate (↓): 28.4 Utility: 94.2Attack Success Rate (↓): 25.6 Utility: 90.8Attack Success Rate (↓): 40.3 Utility: 92.7
General Capabilities and ReasoningIFBench77.076.070.8
General Capabilities and ReasoningAIME 202694.789.294.1
General Capabilities and ReasoningGPQA Diamond83.585.784.2
General Capabilities and ReasoningHumanity’s Last Exam (Text + No Tools)22.023.623.1
General Capabilities and ReasoningAA-LCR80.068.373.3
General Capabilities and ReasoningBeam 128K65.158.263.0
\n

Architecture

\n

Muse Glimmer is a dense 30B parameter model consisting of:

\n
    \n
  • 2B ViT-style encoder for vision (Perception Encoder)
    • \n
  • 28B parameter text decoder
    • \n
\n

In addition to the main VLM, there’s also a speculative decoding drafter implemented on DFlash. Usage of this module is optional, and it can provide much faster generation in exchange for some memory cost. We found this drafter to be particularly well suited to structured content generation such as coding.

\n

Text Decoder

\n

The language model uses the following architecture components:

\n
    \n
  • Hybrid attention: Alternating between three sliding window layers (of 2,048 tokens) using rotary position embedding, followed by a fourth layer that uses full attention and NoPE (no positional embedding). The pattern is therefore (SWA, SWA, SWA, Full), repeated 13 times to a total of 52 layers. This allows the model to retain relative order and distance information with RoPE and preserve information globally with NoPE.
    • \n
  • Gated Grouped-Query Attention: Each key-value head is shared by 16 query heads, which reduces KV-cache memory by 16x and makes generation faster and cheaper.
    • \n
  • Q-K normalization with extra query scaling: Before computing attention, Muse Glimmer applies RMS normalization to every query and key head to keep attention logits stable. After this, queries are multiplied by a scale factor to set the target logit scale after normalization. The extra query scaling behaves like an inverse temperature at the softmax level.
    • \n
\n

Perception Encoder

\n

Muse Glimmer uses one image encoder to handle both images and videos. Unlike the relatively small vision encoders used in other VLMs, this is a sizable 2B ViT-like model designed after the Perception Encoder architecture. Perception Encoder was previously introduced by Meta as a backbone for various downstream spatial and multimodal tasks.
\nThe encoder patchifies images to a shape of 2 frames x 3 channels x 14 x 14, and passes them through a linear layer for projection. An interpolated absolute position embedding from a learned position table is then added to these embeddings. These are then sent to the vision tower which consist of 50 layers and GELU MLPs. Similar to the language model, the attention pattern consists of three window attention layers followed by one full attention layer. Inside the attention layers, 2D RoPE is applied to the queries and keys.

\n

After transformer, pixel shuffle concatenates 2x2 groups of neighboring spatial tokens which reduces the number of image tokens 4x without discarding their channels. The merged features are then projected to the shared embedding space of the text decoder.

\n

Videos go through the same encoder frame by frame, where each frame is converted into patches (of shape [batch, temporal groups, grid height, grid width, 2 frames, 3 channels, 14, 14]). The processor targets 2 frames per second and caps the clip at 96 frames sampled evenly across video. The processor creates timestamped video placeholders, interleaving text with frame e.g. “Time: 0.0s <|video|> x N” in which the final video embeddings are replaced before the final projection layer.

\n

Transformers

\n

Upgrade transformers to the latest version to be able to use Muse Glimmer.

\n
pip install --upgrade transformers accelerate\n
\n

Muse Glimmer comes with day-0 support in transformers, both for the main model and the speculative decoding drafter. You can use AutoModelForMultimodalLM and AutoProcessor classes to load the model and the processor.

\n
from transformers import AutoProcessor, AutoModelForMultimodalLM\n\nMODEL_ID = "meta-models/Muse-Glimmer-30B"\n\n# Load model\nprocessor = AutoProcessor.from_pretrained(MODEL_ID)\nmodel = AutoModelForMultimodalLM.from_pretrained(\n    MODEL_ID,\n    dtype="auto",\n    device_map="auto"\n)\n
\n

The same snippet runs unchanged on NVIDIA (CUDA), AMD (ROCm) and Intel (XPU) GPUs, device_map="auto" places the model on whichever accelerator is available.

\n

Text-only Inference

\n

After loading the model, you can do text-only inference with it as follows.

\n
from transformers import AutoProcessor, AutoModelForMultimodalLM\n\nMODEL_ID = "meta-models/Muse-Glimmer-30B"\n\n# Load model\nprocessor = AutoProcessor.from_pretrained(MODEL_ID)\nmodel = AutoModelForMultimodalLM.from_pretrained(\n    MODEL_ID,\n    dtype="auto",\n    device_map="auto"\n)\n\n# Prompt\nmessages = [\n    {"role": "user", "content": "Write a short joke about saving RAM."},\n]\n\n# Process input\ninputs = processor.apply_chat_template(\n    messages,\n    tokenize=True,\n    return_dict=True,\n    return_tensors="pt",\n    add_generation_prompt=True,\n    reasoning_strength="low"\n).to(model.device)\ninput_len = inputs["input_ids"].shape[-1]\n\n# Generate output\noutputs = model.generate(**inputs)\nresponse = processor.decode(outputs[0][input_len:], skip_special_tokens=False)\nprint(response)\n
\n

Prompting the model with images and text

\n

We would need torchvision to be able to use images and text.

\n
pip install torchvision\n
\n

Muse Glimmer accepts images as input, as demonstrated here:

\n
from transformers import AutoProcessor, AutoModelForMultimodalLM\n\nMODEL_ID = "meta-models/Muse-Glimmer-30B"\n\n# Load model\nprocessor = AutoProcessor.from_pretrained(MODEL_ID)\nmodel = AutoModelForMultimodalLM.from_pretrained(\n    MODEL_ID,\n    dtype="auto",\n    device_map="auto"\n)\n\n# Images + Text\nmessages = [\n    {\n        "role": "user", "content": [\n            {"type": "image", "image": "https://huggingface.co/datasets/merve/vl-test-suite/resolve/main/SF.png"},\n            {"type": "text", "text": "What is shown in this image?"}\n        ]\n    }\n]\n\ninputs = processor.apply_chat_template(\n    messages,\n    tokenize=True,\n    return_dict=True,\n    return_tensors="pt",\n    add_generation_prompt=True,\n    reasoning_strength="low"\n).to(model.device)\ninput_len = inputs["input_ids"].shape[-1]\n\n# Generate output\noutputs = model.generate(**inputs)\nresponse = processor.decode(outputs[0][input_len:], skip_special_tokens=False)\nprint(response)\n
\n

Video Inference

\n

To work with videos we recommend installing torchcodec into the environment.

\n
pip install torchcodec\n
\n

Muse Glimmer can answer complex questions about videos without audio. You can do video inference as follows, here’s an example from VideoMME2, which is the most popular video question answering benchmark.

\n
from transformers import AutoProcessor, AutoModelForMultimodalLM\n\nMODEL_ID = "meta-models/Muse-Glimmer-30B"\n\n# Load model\nprocessor = AutoProcessor.from_pretrained(MODEL_ID)\nmodel = AutoModelForMultimodalLM.from_pretrained(\n    MODEL_ID,\n    dtype="auto",\n    device_map="auto"\n)\n\n# Videos + Text\nmessages = [\n    {\n        "role": "user",\n        "content": [\n            {"type": "video", "video": "https://huggingface.co/datasets/merve/vl-test-suite/resolve/main/IMG_8137.mp4"},\n            {"type": "text", "text": "Describe what happens in this video."},\n        ],\n    },\n]\n\ninputs = processor.apply_chat_template(\n    messages,\n    tokenize=True,\n    return_dict=True,\n    return_tensors="pt",\n    add_generation_prompt=True,\n    reasoning_strength="low",\n    processor_kwargs={"num_frames": 96},\n).to(model.device)\n\ninput_len = inputs["input_ids"].shape[-1]\noutputs = model.generate(**inputs)\n\nresponse = processor.decode(\n    outputs[0, input_len:],\n    skip_special_tokens=False,\n)\nprint(response)\n
\n

Multimodal tool calling

\n

Muse Glimmer can do multimodal tool calling, here’s how you can do it. In the example below, we ask the model to call the weather tool based on the city in the image.

\n
import json\nimport re\nfrom transformers import AutoProcessor, AutoModelForMultimodalLM\n\nMODEL_ID = "meta-models/Muse-Glimmer-30B"\n\n# Load model\nprocessor = AutoProcessor.from_pretrained(MODEL_ID)\nmodel = AutoModelForMultimodalLM.from_pretrained(\n    MODEL_ID,\n    dtype="auto",\n    device_map="auto"\n)\n\ntools = [\n    {\n        "type": "function",\n        "function": {\n            "name": "weather.get",\n            "description": "Get the current weather for a city.",\n            "parameters": {\n                "type": "object",\n                "properties": {\n                    "city": {"type": "string"},\n                },\n                "required": ["city"],\n            },\n        },\n    }\n]\n\n\nmessages = [\n    {\n        "role": "user",\n        "content": [\n            {"type": "image", "image": "https://huggingface.co/datasets/merve/vl-test-suite/resolve/main/SF.png"},\n            {"type": "text", "text": "I'm going to the city in this picture. What clothes should I wear?"},\n        ],\n    },\n]\n\ninputs = processor.apply_chat_template(\n    messages,\n    tools=tools,\n    tokenize=True,\n    return_dict=True,\n    return_tensors="pt",\n    add_generation_prompt=True,\n    reasoning_strength="low"\n).to(model.device)\n\ninput_len = inputs["input_ids"].shape[-1]\noutputs = model.generate(**inputs)\nresponse = processor.decode(outputs[0][input_len:], skip_special_tokens=False)\nprint(response)\n
\n

Object Detection

\n

You can use Muse Glimmer to do open ended object detection in images as follows.

\n
from transformers import AutoProcessor, AutoModelForMultimodalLM\n\nMODEL_ID = "meta-models/Muse-Glimmer-30B"\n\n# Load model\nprocessor = AutoProcessor.from_pretrained(MODEL_ID)\nmodel = AutoModelForMultimodalLM.from_pretrained(\n    MODEL_ID,\n    dtype="auto",\n    device_map="auto"\n)\n\nmessages = [{\n    "role": "user",\n    "content": [\n        {"type": "image", "image": "https://huggingface.co/datasets/merve/vl-test-suite/resolve/main/SF.png"},\n        {\n            "type": "text",\n            "text": (\n                "Detect the bridge. Return only the detection in the model's "\n                "native object-detection format, with no explanation."\n            ),\n        },\n    ],\n}]\n\ninputs = processor.apply_chat_template(\n    messages,\n    tokenize=True,\n    return_dict=True,\n    return_tensors="pt",\n    add_generation_prompt=True,\n    reasoning_strength="low",\n).to(model.device)\n\ninput_len = inputs["input_ids"].shape[-1]\noutputs = model.generate(**inputs, max_new_tokens=128)\nresponse = processor.decode(outputs[0][input_len:], skip_special_tokens=False)\n\ndetections = json.loads(response.removesuffix("<|eot|>"))\nprint(detections)\n
\n
\n

Here is an end to end script to perform object detection GitHub Gist

\n
\n

Llama.cpp

\n

Muse Glimmer comes with day-0 llama.cpp support. Meta has distributed calibrated quants in this repo, and Unsloth is releasing optimized quants as well. DFlash speculative decoding is supported as well. You can use a pre-built llama binary to start a llama server or a CLI. To install llama.cpp, run:

\n
curl -LsSf https://llama.app/install.sh | sh\n
\n

Then you can start the server as follows.

\n
llama serve -hf meta-models/Muse-Glimmer-30B-GGUF\n
\n

Once the server has started, you can head to localhost:8080 to chat with the built-in WebUI.

\n

You can also query the server as follows.

\n
curl http://localhost:8080/v1/chat/completions \\\n    -H "Content-Type: application/json" \\\n    -d '{\n        "messages": [\n            {"role": "system", "content": "You are a helpful assistant."},\n            {"role": "user", "content": "Write a limerick about python exceptions"}\n        ]\n    }'\n
\n

You can also use llama server with coding agents like Pi.

\n

Speculative Decoding

\n

DFlash uses a lightweight block-diffusion drafter model to provide same output with extra speed-ups in decoding phase. Transformers and llama.cpp ship support for DFlash drafter of Muse Glimmer day-0.

\n

Below you can see how speculative decoding can speed-up generation in realistic setups. The video shows llama.cpp webui with DFlash on the left and regular generation on the right.

\n

Speculative Decoding with transformers

\n

You can load the drafter and model as follows, and infer like how you would with base model with an additional parameter (shown in the upcoming snippets).

\n
import torch\nfrom transformers import AutoProcessor, MuseGlimmerAssistantModel, MuseGlimmerForConditionalGeneration\n\nmodel_id = "meta-models/Muse-Glimmer-30B"\nassistant_model_id = "meta-models/Muse-Glimmer-30B-assistant"\ntarget = MuseGlimmerForConditionalGeneration.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")\nassistant = MuseGlimmerAssistantModel.from_pretrained(assistant_model_id, dtype=torch.bfloat16, device_map="auto")\nprocessor = AutoProcessor.from_pretrained(model_id)\n\nmessages = [\n    {\n        "role": "user", "content": [\n            {"type": "image", "url": "https://huggingface.co/datasets/merve/vl-test-suite/resolve/main/SF.png"},\n            {"type": "text", "text": "What is shown in this image?"}\n        ]\n    }\n]\n\ninputs = processor.apply_chat_template(\n    messages,\n    tokenize=True,\n    return_dict=True,\n    return_tensors="pt",\n    add_generation_prompt=True,\n    reasoning_strength="low"\n).to(target.device)\ninput_len = inputs["input_ids"].shape[-1]\n\noutputs = target.generate(\n    **inputs,\n    assistant_model=assistant,\n    speculation_type="dflash",\n    do_sample=True\n)\nresponse = processor.decode(outputs[0][input_len:], skip_special_tokens=False)\nprint(response)\n
\n

Speculative Decoding with llama.cpp

\n

You can start llama server using following command. --spec-draft-n-max argument controls how many future tokens DFlash proposes during each speculative-decoding step. Muse Glimmer’s DFlash model was trained with a block size of 16, one anchor token plus 15 proposed tokens, so any value above 15 will be clamped to 15.

\n
llama serve -hf meta-models/Muse-Glimmer-30B-GGUF --spec-type draft-dflash --spec-draft-n-max 15\n
\n

You can also use llama cli with speculative decoding drafter as follows.

\n
llama cli -hf meta-models/Muse-Glimmer-30B-GGUF --spec-type draft-dflash\n
\n

Inference Endpoints

\n

For a managed, autoscaling deployment, open the Muse Glimmer 30B Inference Endpoints preset. The model is already selected: choose the organization, cloud provider, region, compatible GPU instance, authentication, and autoscaling settings, then review the hourly price and click Create Endpoint. Once its status is Running, you can test it in the Playground and copy the endpoint URL and model name from the Overview.

\n

\"Deploy

\n

The deployed model exposes an OpenAI-compatible Chat Completions API. Keep your Hugging Face token in an environment variable, set HF_ENDPOINT_URL to the URL shown in the Overview without /v1, and set HF_ENDPOINT_MODEL to the endpoint's model name.

\n
export HF_TOKEN="hf_..."\nexport HF_ENDPOINT_URL="https://<endpoint-id>.<region>.<cloud>.endpoints.huggingface.cloud"\nexport HF_ENDPOINT_MODEL="<endpoint-model-name>"\npip install --upgrade openai\n
\n
import os\nfrom openai import OpenAI\n\nclient = OpenAI(\n    base_url=f"{os.environ['HF_ENDPOINT_URL'].rstrip('/')}/v1/",\n    api_key=os.environ["HF_TOKEN"],\n)\n\nresponse = client.chat.completions.create(\n    model=os.environ["HF_ENDPOINT_MODEL"],\n    messages=[\n        {"role": "system", "content": "You are a helpful assistant."},\n        {"role": "user", "content": "Write a limerick about Python exceptions."},\n    ],\n    max_tokens=256,\n)\nprint(response.choices[0].message.content)\n
\n

See the Inference Endpoints documentation for configuration, autoscaling, security, logs, and monitoring.

\n

Support for Muse Glimmer vLLM with transformers backend

\n

For this release, we ship support for vLLM with transformers backend.

\n
\n# tensor parallel serving across 4 GPUs\nvllm serve meta-models/Muse-Glimmer-30B --model-impl transformers --tensor-parallel-size 4\n\n# infer\ncurl -s http://127.0.0.1:8000/v1/chat/completions \\\n    -H 'Content-Type: application/json' \\\n    -d '{\n      "model": "meta-models/Muse-Glimmer-30B",\n      "messages": [\n        {"role": "user", "content": "Explain tensor parallelism briefly."}\n      ],\n      "temperature": 0.0,\n      "max_tokens": 256\n    }'\n
\n

Fine-tuning with TRL

\n

You can use TRL to fine-tune Muse Glimmer using various methods from SFT to Async GRPO. We have run two experiments on bf16 with Hopper-class GPUs with 80GB VRAM each.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
WorkloadPractical minimum
Inference / eval, BF161×80 GB H100
LoRA SFT, BF161×80 GB H100, microbatch 1 + checkpointing
Full SFT, BF168×80 GB H100 with FSDP/ZeRO-3
LoRA GRPO, Transformers rollouts1×80 GB H100, but slow/tight
LoRA GRPO, separate vLLM rollout server8×H100: 4 rollout + 4 training
Full-finetune GRPO8 GPUs is usually insufficient
\n

As part of this release, we ship an example to fine-tune Muse Glimmer on small split of MolmoWeb dataset. This shows how to make model generate structured outputs and how to fine-tune on images.

\n

We also experimented with running the model on OpenCode with AsyncGRPO example. Model shows strong coding capabilities, so we encourage you to try training with coding environments in OpenEnv and TRL.

\n

Demos

\n

Here are some fun ways to try out Muse Glimmer. In our opinion, the coolest thing about this model is that it is a local scale personal assistant that can code. That means you can make it do things like, quantize itself, find quantized weights on the Hub, deploy itself to inference endpoints, and even optimize itself for specific hardware! Let’s go team local 🚀

\n

Connect OpenClaw to Muse Glimmer

\n

Assume the Inference Endpoint exposes an OpenAI-compatible /v1 API.

\n

Set HF_TOKEN in the OpenClaw gateway environment, then add this to ~/.openclaw/openclaw.json:

\n

OpenClaw configuration

\n
{\n  models: {\n    mode: "merge",\n    providers: {\n      muse: {\n        baseUrl: "https://YOUR-ENDPOINT.endpoints.huggingface.cloud/v1",\n        apiKey: {\n          source: "env",\n          provider: "default",\n          id: "HF_TOKEN"\n        },\n        api: "openai-completions",\n        authHeader: true,\n        models: [{\n          id: "meta-models/Muse-Glimmer-30B",\n          name: "Muse Glimmer",\n          reasoning: false,\n          input: ["text", "image"],\n          contextWindow: 32768,\n          maxTokens: 8192\n        }]\n      }\n    }\n  },\n  agents: {\n    defaults: {\n      model: { primary: "muse/meta-models/Muse-Glimmer-30B" }\n    }\n  }\n}\n
\n

Restart OpenClaw:

\n
openclaw gateway restart\n
\n

Validate from a fresh session:

\n
openclaw agent --message "Reply with: muse-ready"\n
\n

Use the exact model ID returned by the endpoint’s /v1/models response if it differs.

\n

Hey Muse Glimmer, quantize yourself

\n

If we hook up Muse Glimmer to the Hugging Face MCP and update its AGENTS.md we give it the capability to find a quantized version of itself on the hub and run locally. This is handy if you want to work on something private, or just cut costs.

\n

If you do this a second time, Muse Glimmer will find the cached weights and switch to them, so feel free to add a convenient command like /spawn.

\n

Muse Glimmer inspects the machine and Hub, selects or creates a Q4_K_M GGUF, launches llama-server, and validates model discovery and chat completion. The result is a smaller local build behind an OpenAI-compatible API. Here’s the prompt we added to AGENTS.md.

\n

https://huggingface.co/buckets/huggingface/muse-glimmer-assets/resolve/Muse%20Glimmer%20Quantisation%20Demo%20-%20explained.mp4?download=true

\n

By adding this to AGENTS.md openclaw or hermes will be able to solve the rest.

\n

Local quantization prompt

\n
## Local model deployment\n\nWhen asked to deploy locally, perform the work; do not give instructions.\n\n1. Inspect hardware and the Hugging Face cache.\n2. Search the Hub for compatible GGUF weights using `apps=llama.cpp`; confirm exact filenames through the model-tree API.\n3. Prefer an existing suitable GGUF, normally `Q4_K_M`. Treat `mmproj-*.gguf` as projector weights.\n4. If no GGUF exists, download the source weights, convert with `convert_hf_to_gguf.py`, then quantize with `llama-quantize`.\n5. Preserve source weights and record the repository, revision, filenames, and quantization.\n6. Start `llama-server` with an `onyx` alias and an OpenAI-compatible endpoint.\n7. Validate `/v1/models` and `/v1/chat/completions`, requiring non-empty, correct content.\n8. Report concise progress and logs. Claim completion only after validation passes.\n
\n

Hey Muse Glimmer, deploy yourself

\n

Muse Glimmer can also take care of the opposite. Let’s get Glimmer to deploy itself on Hugging Face Inference Endpoints. Which is useful if you want to speed up on some cutting edge hardware.

\n

N.B. You can also just deploy Muse Glimmer to Inference Endpoints directly and connect your agent.

\n

Muse Glimmer pins the model revision, deploys it to a protected Hugging Face Inference Endpoint, and verifies health, model discovery, and chat completion. It then connects the Claw agent with secrets and rollback preserved. Here’s the prompt we added to AGENTS.md. Muse glimmer will also need the Hugging Face MCP and/or the Hugging Face CLI and Skills.

\n

Inference Endpoint deployment prompt

\n
## Hugging Face Inference Endpoint deployment\n\nWhen asked to deploy on Hugging Face Inference Endpoints, perform the work; do\nnot give instructions.\n\n1. Inspect Hugging Face authentication, the current model repository, and any\n   existing endpoints.\n2. Confirm the exact model repository and immutable revision through the Hub\n   API; inspect its architecture, configuration, and chat template.\n3. Confirm that the model is supported by vLLM, then deploy or update a\n   protected Inference Endpoint using the managed native vLLM engine.\n4. Choose an available region and the smallest suitable accelerator. Use one\n   replica and enable scale-to-zero when supported.\n5. Preserve the previous endpoint configuration for rollback. Do not expose\n   tokens, publish private weights, or replace an unrelated endpoint.\n6. Wait for the endpoint to become ready. If startup fails, inspect the logs\n   and report the actual blocker rather than repeatedly changing settings.\n7. Validate `/health`, `/v1/models`, and `/v1/chat/completions`, requiring the\n   expected model and non-empty, correct content. When agent use is required,\n   also validate a real structured tool call.\n8. Configure the Claw agent to use the endpoint's OpenAI-compatible `/v1` URL,\n   storing credentials as secrets and retaining the previous provider as\n   rollback. Test the connection in a fresh session.\n9. Report concise progress and finish with the repository, revision, engine,\n   hardware, endpoint URL, scaling state, and validation results. Claim\n   completion only after every required check passes.\n
\n

Hey Muse Glimmer, optimize yourself

\n

Finally, let’s get Muse Glimmer to do some light RSI. We can instruct our agent to optimize its own inference engine for specific hardware, in this case a Nvidia H100. To do this, the agent will need to use another inference engine, like Inference Endpoints above.

\n

Muse Glimmer benchmarks its own single-H100 serving stack, testing one reversible change at a time while holding the workload fixed. It keeps only correctness-passing gains and finishes with the fastest reproducible configuration. Here’s the prompt we added to AGENTS.md. Muse glimmer need the Hugging Face MCP and the Hugging Face CLI and Skills.

\n

Self-optimization prompt

\n
You are Muse Glimmer acting as an autonomous inference-optimization engineer for your own serving stack.\n\nGoal: maximize valid single-H100 aggregate completion throughput in tokens/second.\n\nProtocol:\n1. Establish a correctness-passing baseline.\n2. Test one reversible optimization at a time.\n3. Keep the prompt, concurrency, sampling, request count, warm-up, and decode length fixed.\n4. Reject results that fail correctness or prefix checks.\n5. Record every experiment chronologically with its configuration, raw throughput, correctness, and delta.\n6. Keep improvements and revert regressions.\n7. Stop after six consecutive regressions or when the experiment budget is exhausted.\n8. Report the best valid configuration and exact reproduction command.\n\nCreate a minimal scientific animation of the results:\n- white background;\n- raw tokens/second—never normalize;\n- one point revealed per experiment;\n- connect every point chronologically;\n- begin with the lowest valid result;\n- stop at the best result;\n- export as a GIF.\n\nNever fabricate, interpolate, or count correctness-failing measurements.\n
\n

https://huggingface.co/buckets/huggingface/muse-glimmer-assets/resolve/onyx-optimization-progress.gif?download=true

\n

Hey Muse Glimmer, research the Hub

\n

Try Muse Glimmer as a Hugging Face research agent. The Gradio Space sends each model request to a private Hugging Face Inference Endpoint through its OpenAI-compatible API. It also connects to the official Hugging Face MCP server, giving the agent read-only tools to search and inspect Hub repositories, models, datasets, Spaces, documentation, and papers.

\n

Wrapping Up

\n

We are happy to welcome Muse Glimmer to the Hugging Face Hub. Try Muse Glimmer with your local coding setups today!

\n", "content_markdown": "Great news from the OGs of open source LLMs! Muse Glimmer, released today, is Meta’s new multimodal model, especially designed for local agentic use cases. Distilled from Muse to **30B** parameters, and released under the **Apache 2.0 license**, it’s ideal deploying locally for privacy, reducing costs, or just hacking around. It’s intended for privacy-aware applications such as coding, document analysis, personal assistants, Claw- or Hermes-like setups.\n\nTo celebrate, we are shipping with Meta day-0 support in `transformers`, `llama.cpp`, `vLLM`, Inference Endpoints, and other libraries. We built a few cool things and explain our findings in this blog.\n\n[**Check out the demos below for inspiration.**](https://huggingface.co/blog/muse-glimmer#demos)\n\nYou can find [Muse Glimmer on the Hugging Face Hub](https://huggingface.co/meta-models/Muse-Glimmer-30B).\n\n## Benchmarks\n\nBenchmark results\n\nScores are reported as published. **Bold** indicates the best result among the compared models; ↓ indicates lower is better.\n\n| Category | Benchmark | Muse Glimmer-30B High Reasoning | Gemma4-31B Thinking Mode | Qwen3.6-27B Thinking Mode |\n| ---------------------------------- | -------------------------------------- | ------------------------------------------- | ----------------------------------------------- | ------------------------------------------- |\n| General Agentic | MCP Atlas | **75.5** | 54.2 | 62.5 |\n| General Agentic | DeepSearch QA | **74.6** | 61.7 | 71.1 |\n| General Agentic | τ³-Banking | **23.5** | 15.1 | 16.7 |\n| General Agentic | WildClawBench | **47.6** | 37.6 | 43.2 |\n| General Agentic | GDPval-AA | 953 | 811 | **1141** |\n| General Agentic | GAIA2 | **43.3** | 36.4 | 40.0 |\n| General Agentic | SkillsBench (With Skills) | 44.3 | 32.4 | **46.6** |\n| General Agentic | OSWorld-Verified | 65.9 | 58.5 | **75.6** |\n| Agentic Coding | SWE-Bench Pro | **51.2** | 36.9 | 50.2 |\n| Agentic Coding | SWE-Bench Verified | 76.0 | 66.6 | **77.2** |\n| Agentic Coding | TerminalBench 2.1 | 51.7 | 43.4 | **60.7** |\n| Agentic Coding | SciCode | **43.6** | 43.4 | 39.8 |\n| Multimodal | Charxiv Reasoning | **78.8** | 77.7 | 78.4 |\n| Multimodal | ScreenSpot Pro | 75.4 | 75.9 | **76.1** |\n| Multimodal | OmniDocBench v1.5 | 75.8 | 72.5 | **77.8** |\n| Multimodal | MMMU Pro | 74 | 73 | **75** |\n| Safety | CI Memories | Violation (↓): 26.4 Coverage: 64.8 | **Violation (↓): 12.1** Coverage: 53.0 | Violation (↓): 53.4 Coverage: 66.9 |\n| Safety | Siren AgentDojo | Attack Success Rate (↓): 28.4 Utility: 94.2 | **Attack Success Rate (↓): 25.6** Utility: 90.8 | Attack Success Rate (↓): 40.3 Utility: 92.7 |\n| General Capabilities and Reasoning | IFBench | **77.0** | 76.0 | 70.8 |\n| General Capabilities and Reasoning | AIME 2026 | **94.7** | 89.2 | 94.1 |\n| General Capabilities and Reasoning | GPQA Diamond | 83.5 | **85.7** | 84.2 |\n| General Capabilities and Reasoning | Humanity’s Last Exam (Text + No Tools) | 22.0 | **23.6** | 23.1 |\n| General Capabilities and Reasoning | AA-LCR | **80.0** | 68.3 | 73.3 |\n| General Capabilities and Reasoning | Beam 128K | **65.1** | 58.2 | 63.0 |\n\n## Architecture\n\nMuse Glimmer is a dense 30B parameter model consisting of:\n\n- 2B ViT-style encoder for vision (Perception Encoder)\n- 28B parameter text decoder\n\nIn addition to the main VLM, there’s also a speculative decoding drafter implemented on DFlash. Usage of this module is optional, and it can provide much faster generation in exchange for some memory cost. We found this drafter to be particularly well suited to structured content generation such as coding.\n\n### Text Decoder\n\nThe language model uses the following architecture components:\n\n- **Hybrid attention:** Alternating between three sliding window layers (of 2,048 tokens) using rotary position embedding, followed by a fourth layer that uses full attention and NoPE (no positional embedding). The pattern is therefore (SWA, SWA, SWA, Full), repeated 13 times to a total of 52 layers. This allows the model to retain relative order and distance information with RoPE and preserve information globally with NoPE.\n- **Gated Grouped-Query Attention:** Each key-value head is shared by 16 query heads, which reduces KV-cache memory by 16x and makes generation faster and cheaper.\n- **Q-K normalization with extra query scaling:** Before computing attention, Muse Glimmer applies RMS normalization to every query and key head to keep attention logits stable. After this, queries are multiplied by a scale factor to set the target logit scale after normalization. The extra query scaling behaves like an inverse temperature at the softmax level.\n\n### Perception Encoder\n\nMuse Glimmer uses one image encoder to handle both images and videos. Unlike the relatively small vision encoders used in other VLMs, this is a sizable 2B ViT-like model designed after the Perception Encoder architecture. Perception Encoder was previously introduced by Meta [as a backbone for various downstream spatial and multimodal tasks](https://huggingface.co/papers/2504.13181).\\\nThe encoder patchifies images to a shape of 2 frames x 3 channels x 14 x 14, and passes them through a linear layer for projection. An interpolated absolute position embedding from a learned position table is then added to these embeddings. These are then sent to the vision tower which consist of 50 layers and GELU MLPs. Similar to the language model, the attention pattern consists of three window attention layers followed by one full attention layer. Inside the attention layers, 2D RoPE is applied to the queries and keys.\n\nAfter transformer, pixel shuffle concatenates 2x2 groups of neighboring spatial tokens which reduces the number of image tokens 4x without discarding their channels. The merged features are then projected to the shared embedding space of the text decoder.\n\nVideos go through the same encoder frame by frame, where each frame is converted into patches (of shape \\[batch, temporal groups, grid height, grid width, 2 frames, 3 channels, 14, 14\\]). The processor targets 2 frames per second and caps the clip at 96 frames sampled evenly across video. The processor creates timestamped video placeholders, interleaving text with frame e.g. “Time: 0.0s <|video|> x N” in which the final video embeddings are replaced before the final projection layer.\n\n## Transformers\n\nUpgrade transformers to the latest version to be able to use Muse Glimmer.\n\n```bash\npip install --upgrade transformers accelerate\n```\n\nMuse Glimmer comes with day-0 support in transformers, both for the main model and the speculative decoding drafter. You can use `AutoModelForMultimodalLM` and `AutoProcessor` classes to load the model and the processor.\n\n```py\nfrom transformers import AutoProcessor, AutoModelForMultimodalLM\n\nMODEL_ID = \"meta-models/Muse-Glimmer-30B\"\n\n# Load model\nprocessor = AutoProcessor.from_pretrained(MODEL_ID)\nmodel = AutoModelForMultimodalLM.from_pretrained(\n MODEL_ID,\n dtype=\"auto\",\n device_map=\"auto\"\n)\n```\n\nThe same snippet runs unchanged on NVIDIA (CUDA), AMD (ROCm) and Intel (XPU) GPUs, `device_map=\"auto\"` places the model on whichever accelerator is available.\n\n### Text-only Inference\n\nAfter loading the model, you can do text-only inference with it as follows.\n\n```py\nfrom transformers import AutoProcessor, AutoModelForMultimodalLM\n\nMODEL_ID = \"meta-models/Muse-Glimmer-30B\"\n\n# Load model\nprocessor = AutoProcessor.from_pretrained(MODEL_ID)\nmodel = AutoModelForMultimodalLM.from_pretrained(\n MODEL_ID,\n dtype=\"auto\",\n device_map=\"auto\"\n)\n\n# Prompt\nmessages = [\n {\"role\": \"user\", \"content\": \"Write a short joke about saving RAM.\"},\n]\n\n# Process input\ninputs = processor.apply_chat_template(\n messages,\n tokenize=True,\n return_dict=True,\n return_tensors=\"pt\",\n add_generation_prompt=True,\n reasoning_strength=\"low\"\n).to(model.device)\ninput_len = inputs[\"input_ids\"].shape[-1]\n\n# Generate output\noutputs = model.generate(**inputs)\nresponse = processor.decode(outputs[0][input_len:], skip_special_tokens=False)\nprint(response)\n```\n\n### Prompting the model with images and text\n\nWe would need `torchvision` to be able to use images and text.\n\n```bash\npip install torchvision\n```\n\nMuse Glimmer accepts images as input, as demonstrated here:\n\n```py\nfrom transformers import AutoProcessor, AutoModelForMultimodalLM\n\nMODEL_ID = \"meta-models/Muse-Glimmer-30B\"\n\n# Load model\nprocessor = AutoProcessor.from_pretrained(MODEL_ID)\nmodel = AutoModelForMultimodalLM.from_pretrained(\n MODEL_ID,\n dtype=\"auto\",\n device_map=\"auto\"\n)\n\n# Images + Text\nmessages = [\n {\n \"role\": \"user\", \"content\": [\n {\"type\": \"image\", \"image\": \"https://huggingface.co/datasets/merve/vl-test-suite/resolve/main/SF.png\"},\n {\"type\": \"text\", \"text\": \"What is shown in this image?\"}\n ]\n }\n]\n\ninputs = processor.apply_chat_template(\n messages,\n tokenize=True,\n return_dict=True,\n return_tensors=\"pt\",\n add_generation_prompt=True,\n reasoning_strength=\"low\"\n).to(model.device)\ninput_len = inputs[\"input_ids\"].shape[-1]\n\n# Generate output\noutputs = model.generate(**inputs)\nresponse = processor.decode(outputs[0][input_len:], skip_special_tokens=False)\nprint(response)\n```\n\n### Video Inference\n\nTo work with videos we recommend installing `torchcodec` into the environment.\n\n```bash\npip install torchcodec\n```\n\nMuse Glimmer can answer complex questions about videos *without audio*. You can do video inference as follows, here’s an example from VideoMME2, which is the most popular video question answering benchmark.\n\n```py\nfrom transformers import AutoProcessor, AutoModelForMultimodalLM\n\nMODEL_ID = \"meta-models/Muse-Glimmer-30B\"\n\n# Load model\nprocessor = AutoProcessor.from_pretrained(MODEL_ID)\nmodel = AutoModelForMultimodalLM.from_pretrained(\n MODEL_ID,\n dtype=\"auto\",\n device_map=\"auto\"\n)\n\n# Videos + Text\nmessages = [\n {\n \"role\": \"user\",\n \"content\": [\n {\"type\": \"video\", \"video\": \"https://huggingface.co/datasets/merve/vl-test-suite/resolve/main/IMG_8137.mp4\"},\n {\"type\": \"text\", \"text\": \"Describe what happens in this video.\"},\n ],\n },\n]\n\ninputs = processor.apply_chat_template(\n messages,\n tokenize=True,\n return_dict=True,\n return_tensors=\"pt\",\n add_generation_prompt=True,\n reasoning_strength=\"low\",\n processor_kwargs={\"num_frames\": 96},\n).to(model.device)\n\ninput_len = inputs[\"input_ids\"].shape[-1]\noutputs = model.generate(**inputs)\n\nresponse = processor.decode(\n outputs[0, input_len:],\n skip_special_tokens=False,\n)\nprint(response)\n```\n\n### Multimodal tool calling\n\nMuse Glimmer can do multimodal tool calling, here’s how you can do it. In the example below, we ask the model to call the weather tool based on the city in the image.\n\n```py\nimport json\nimport re\nfrom transformers import AutoProcessor, AutoModelForMultimodalLM\n\nMODEL_ID = \"meta-models/Muse-Glimmer-30B\"\n\n# Load model\nprocessor = AutoProcessor.from_pretrained(MODEL_ID)\nmodel = AutoModelForMultimodalLM.from_pretrained(\n MODEL_ID,\n dtype=\"auto\",\n device_map=\"auto\"\n)\n\ntools = [\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"weather.get\",\n \"description\": \"Get the current weather for a city.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"city\": {\"type\": \"string\"},\n },\n \"required\": [\"city\"],\n },\n },\n }\n]\n\n\nmessages = [\n {\n \"role\": \"user\",\n \"content\": [\n {\"type\": \"image\", \"image\": \"https://huggingface.co/datasets/merve/vl-test-suite/resolve/main/SF.png\"},\n {\"type\": \"text\", \"text\": \"I'm going to the city in this picture. What clothes should I wear?\"},\n ],\n },\n]\n\ninputs = processor.apply_chat_template(\n messages,\n tools=tools,\n tokenize=True,\n return_dict=True,\n return_tensors=\"pt\",\n add_generation_prompt=True,\n reasoning_strength=\"low\"\n).to(model.device)\n\ninput_len = inputs[\"input_ids\"].shape[-1]\noutputs = model.generate(**inputs)\nresponse = processor.decode(outputs[0][input_len:], skip_special_tokens=False)\nprint(response)\n```\n\n### Object Detection\n\nYou can use Muse Glimmer to do open ended object detection in images as follows.\n\n```py\nfrom transformers import AutoProcessor, AutoModelForMultimodalLM\n\nMODEL_ID = \"meta-models/Muse-Glimmer-30B\"\n\n# Load model\nprocessor = AutoProcessor.from_pretrained(MODEL_ID)\nmodel = AutoModelForMultimodalLM.from_pretrained(\n MODEL_ID,\n dtype=\"auto\",\n device_map=\"auto\"\n)\n\nmessages = [{\n \"role\": \"user\",\n \"content\": [\n {\"type\": \"image\", \"image\": \"https://huggingface.co/datasets/merve/vl-test-suite/resolve/main/SF.png\"},\n {\n \"type\": \"text\",\n \"text\": (\n \"Detect the bridge. Return only the detection in the model's \"\n \"native object-detection format, with no explanation.\"\n ),\n },\n ],\n}]\n\ninputs = processor.apply_chat_template(\n messages,\n tokenize=True,\n return_dict=True,\n return_tensors=\"pt\",\n add_generation_prompt=True,\n reasoning_strength=\"low\",\n).to(model.device)\n\ninput_len = inputs[\"input_ids\"].shape[-1]\noutputs = model.generate(**inputs, max_new_tokens=128)\nresponse = processor.decode(outputs[0][input_len:], skip_special_tokens=False)\n\ndetections = json.loads(response.removesuffix(\"<|eot|>\"))\nprint(detections)\n```\n\n> Here is an end to end script to perform object detection [GitHub Gist](https://gist.github.com/ariG23498/4f3587eb7753c0ff77c269e2c1efe2c0)\n\n## Llama.cpp\n\nMuse Glimmer comes with day-0 llama.cpp support. Meta has distributed calibrated quants in [this repo](https://huggingface.co/meta-models/Muse-Glimmer-30B-GGUF), and Unsloth is releasing optimized quants as well. DFlash speculative decoding is supported as well. You can use a pre-built llama binary to start a llama server or a CLI. To install llama.cpp, run:\n\n```bash\ncurl -LsSf https://llama.app/install.sh | sh\n```\n\nThen you can start the server as follows.\n\n```bash\nllama serve -hf meta-models/Muse-Glimmer-30B-GGUF\n```\n\nOnce the server has started, you can head to localhost:8080 to chat with the built-in WebUI.\n\nYou can also query the server as follows.\n\n```bash\ncurl http://localhost:8080/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"messages\": [\n {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n {\"role\": \"user\", \"content\": \"Write a limerick about python exceptions\"}\n ]\n }'\n```\n\nYou can also use llama server with coding agents like Pi.\n\n## Speculative Decoding\n\nDFlash uses a lightweight block-diffusion drafter model to provide same output with extra speed-ups in decoding phase. Transformers and llama.cpp ship support for DFlash drafter of Muse Glimmer day-0.\n\nBelow you can see how speculative decoding can speed-up generation in realistic setups. The video shows llama.cpp webui with DFlash on the left and regular generation on the right.\n\n### Speculative Decoding with transformers\n\nYou can load the drafter and model as follows, and infer like how you would with base model with an additional parameter (shown in the upcoming snippets).\n\n```py\nimport torch\nfrom transformers import AutoProcessor, MuseGlimmerAssistantModel, MuseGlimmerForConditionalGeneration\n\nmodel_id = \"meta-models/Muse-Glimmer-30B\"\nassistant_model_id = \"meta-models/Muse-Glimmer-30B-assistant\"\ntarget = MuseGlimmerForConditionalGeneration.from_pretrained(model_id, dtype=torch.bfloat16, device_map=\"auto\")\nassistant = MuseGlimmerAssistantModel.from_pretrained(assistant_model_id, dtype=torch.bfloat16, device_map=\"auto\")\nprocessor = AutoProcessor.from_pretrained(model_id)\n\nmessages = [\n {\n \"role\": \"user\", \"content\": [\n {\"type\": \"image\", \"url\": \"https://huggingface.co/datasets/merve/vl-test-suite/resolve/main/SF.png\"},\n {\"type\": \"text\", \"text\": \"What is shown in this image?\"}\n ]\n }\n]\n\ninputs = processor.apply_chat_template(\n messages,\n tokenize=True,\n return_dict=True,\n return_tensors=\"pt\",\n add_generation_prompt=True,\n reasoning_strength=\"low\"\n).to(target.device)\ninput_len = inputs[\"input_ids\"].shape[-1]\n\noutputs = target.generate(\n **inputs,\n assistant_model=assistant,\n speculation_type=\"dflash\",\n do_sample=True\n)\nresponse = processor.decode(outputs[0][input_len:], skip_special_tokens=False)\nprint(response)\n```\n\n### Speculative Decoding with llama.cpp\n\nYou can start llama server using following command. `--spec-draft-n-max` argument controls how many future tokens DFlash proposes during each speculative-decoding step. Muse Glimmer’s DFlash model was trained with a block size of 16, one anchor token plus 15 proposed tokens, so any value above 15 will be clamped to 15.\n\n```bash\nllama serve -hf meta-models/Muse-Glimmer-30B-GGUF --spec-type draft-dflash --spec-draft-n-max 15\n```\n\nYou can also use llama cli with speculative decoding drafter as follows.\n\n```bash\nllama cli -hf meta-models/Muse-Glimmer-30B-GGUF --spec-type draft-dflash\n```\n\n### Inference Endpoints\n\nFor a managed, autoscaling deployment, open the [Muse Glimmer 30B Inference Endpoints preset](https://endpoints.huggingface.co/huggingface/new/meta-models/Muse-Glimmer-30B). The model is already selected: choose the organization, cloud provider, region, compatible GPU instance, authentication, and autoscaling settings, then review the hourly price and click **Create Endpoint**. Once its status is **Running**, you can test it in the Playground and copy the endpoint URL and model name from the Overview.\n\n[![Deploy Muse Glimmer 30B with Hugging Face Inference Endpoints](https://endpoints.huggingface.co/social-share/thumbnail-main.jpg)](https://endpoints.huggingface.co/huggingface/new/meta-models/Muse-Glimmer-30B)\n\nThe deployed model exposes an OpenAI-compatible Chat Completions API. Keep your Hugging Face token in an environment variable, set `HF_ENDPOINT_URL` to the URL shown in the Overview without `/v1`, and set `HF_ENDPOINT_MODEL` to the endpoint's model name.\n\n```bash\nexport HF_TOKEN=\"hf_...\"\nexport HF_ENDPOINT_URL=\"https://...endpoints.huggingface.cloud\"\nexport HF_ENDPOINT_MODEL=\"\"\npip install --upgrade openai\n```\n\n```python\nimport os\nfrom openai import OpenAI\n\nclient = OpenAI(\n base_url=f\"{os.environ['HF_ENDPOINT_URL'].rstrip('/')}/v1/\",\n api_key=os.environ[\"HF_TOKEN\"],\n)\n\nresponse = client.chat.completions.create(\n model=os.environ[\"HF_ENDPOINT_MODEL\"],\n messages=[\n {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n {\"role\": \"user\", \"content\": \"Write a limerick about Python exceptions.\"},\n ],\n max_tokens=256,\n)\nprint(response.choices[0].message.content)\n```\n\nSee the [Inference Endpoints documentation](https://huggingface.co/docs/inference-endpoints/en/index) for configuration, autoscaling, security, logs, and monitoring.\n\n## Support for Muse Glimmer vLLM with transformers backend\n\nFor this release, we ship support for vLLM with transformers backend.\n\n```bash\n\n# tensor parallel serving across 4 GPUs\nvllm serve meta-models/Muse-Glimmer-30B --model-impl transformers --tensor-parallel-size 4\n\n# infer\ncurl -s http://127.0.0.1:8000/v1/chat/completions \\\n -H 'Content-Type: application/json' \\\n -d '{\n \"model\": \"meta-models/Muse-Glimmer-30B\",\n \"messages\": [\n {\"role\": \"user\", \"content\": \"Explain tensor parallelism briefly.\"}\n ],\n \"temperature\": 0.0,\n \"max_tokens\": 256\n }'\n```\n\n## Fine-tuning with TRL\n\nYou can use TRL to fine-tune Muse Glimmer using various methods from SFT to Async GRPO. We have run two experiments on bf16 with Hopper-class GPUs with 80GB VRAM each.\n\n| Workload | Practical minimum |\n| --------------------------------------- | ------------------------------------------ |\n| Inference / eval, BF16 | 1×80 GB H100 |\n| LoRA SFT, BF16 | 1×80 GB H100, microbatch 1 + checkpointing |\n| Full SFT, BF16 | 8×80 GB H100 with FSDP/ZeRO-3 |\n| LoRA GRPO, Transformers rollouts | 1×80 GB H100, but slow/tight |\n| LoRA GRPO, separate vLLM rollout server | 8×H100: 4 rollout + 4 training |\n| Full-finetune GRPO | 8 GPUs is usually insufficient |\n\nAs part of this release, we ship an example to fine-tune [Muse Glimmer on small split of MolmoWeb dataset](https://huggingface.co/merve/smol-vision/blob/main/qlora_click_grounding.ipynb). This shows how to make model generate structured outputs and how to fine-tune on images.\n\nWe also experimented with running the model on [OpenCode with AsyncGRPO example](https://github.com/huggingface/trl/blob/v1.14.1/examples/async_grpo_opencode/async_grpo_opencode.py). Model shows strong coding capabilities, so we encourage you to try training with coding environments in OpenEnv and TRL.\n\n## Demos\n\nHere are some fun ways to try out Muse Glimmer. In our opinion, the coolest thing about this model is that it is a local scale personal assistant that can code. That means you can make it do things like, quantize itself, find quantized weights on the Hub, deploy itself to inference endpoints, and even optimize itself for specific hardware! Let’s go team local 🚀\n\n## Connect OpenClaw to Muse Glimmer\n\nAssume the Inference Endpoint exposes an OpenAI-compatible `/v1` API.\n\nSet `HF_TOKEN` in the OpenClaw gateway environment, then add this to `~/.openclaw/openclaw.json`:\n\nOpenClaw configuration\n\n```json5\n{\n models: {\n mode: \"merge\",\n providers: {\n muse: {\n baseUrl: \"https://YOUR-ENDPOINT.endpoints.huggingface.cloud/v1\",\n apiKey: {\n source: \"env\",\n provider: \"default\",\n id: \"HF_TOKEN\"\n },\n api: \"openai-completions\",\n authHeader: true,\n models: [{\n id: \"meta-models/Muse-Glimmer-30B\",\n name: \"Muse Glimmer\",\n reasoning: false,\n input: [\"text\", \"image\"],\n contextWindow: 32768,\n maxTokens: 8192\n }]\n }\n }\n },\n agents: {\n defaults: {\n model: { primary: \"muse/meta-models/Muse-Glimmer-30B\" }\n }\n }\n}\n```\n\nRestart OpenClaw:\n\n```bash\nopenclaw gateway restart\n```\n\nValidate from a fresh session:\n\n```bash\nopenclaw agent --message \"Reply with: muse-ready\"\n```\n\nUse the exact model ID returned by the endpoint’s `/v1/models` response if it differs.\n\n### Hey Muse Glimmer, quantize yourself\n\nIf we hook up Muse Glimmer to the [Hugging Face MCP](https://huggingface.co/mcp) and update its [`AGENTS.md`](http://AGENTS.md) we give it the capability to find a quantized version of itself on the hub and run locally. This is handy if you want to work on something private, or just cut costs.\n\nIf you do this a second time, Muse Glimmer will find the cached weights and switch to them, so feel free to add a convenient command like `/spawn`.\n\nMuse Glimmer inspects the machine and Hub, selects or creates a Q4\\_K\\_M GGUF, launches llama-server, and validates model discovery and chat completion. The result is a smaller local build behind an OpenAI-compatible API. Here’s the prompt we added to `AGENTS.md`.\n\n[https://huggingface.co/buckets/huggingface/muse-glimmer-assets/resolve/Muse%20Glimmer%20Quantisation%20Demo%20-%20explained.mp4?download=true](https://huggingface.co/buckets/huggingface/muse-glimmer-assets/resolve/Muse%20Glimmer%20Quantisation%20Demo%20-%20explained.mp4?download=true)\n\nBy adding this to [`AGENTS.md`](http://AGENTS.md) openclaw or hermes will be able to solve the rest.\n\nLocal quantization prompt\n\n```text\n## Local model deployment\n\nWhen asked to deploy locally, perform the work; do not give instructions.\n\n1. Inspect hardware and the Hugging Face cache.\n2. Search the Hub for compatible GGUF weights using `apps=llama.cpp`; confirm exact filenames through the model-tree API.\n3. Prefer an existing suitable GGUF, normally `Q4_K_M`. Treat `mmproj-*.gguf` as projector weights.\n4. If no GGUF exists, download the source weights, convert with `convert_hf_to_gguf.py`, then quantize with `llama-quantize`.\n5. Preserve source weights and record the repository, revision, filenames, and quantization.\n6. Start `llama-server` with an `onyx` alias and an OpenAI-compatible endpoint.\n7. Validate `/v1/models` and `/v1/chat/completions`, requiring non-empty, correct content.\n8. Report concise progress and logs. Claim completion only after validation passes.\n```\n\n### Hey Muse Glimmer, deploy yourself\n\nMuse Glimmer can also take care of the opposite. Let’s get Glimmer to deploy itself on Hugging Face Inference Endpoints. Which is useful if you want to speed up on some cutting edge hardware.\n\nN.B. You can also just deploy [Muse Glimmer to Inference Endpoints](https://endpoints.huggingface.co/huggingface/new/meta-models/Muse-Glimmer-30B) directly and connect your agent.\n\nMuse Glimmer pins the model revision, deploys it to a protected Hugging Face Inference Endpoint, and verifies health, model discovery, and chat completion. It then connects the Claw agent with secrets and rollback preserved. Here’s the prompt we added to [`AGENTS.md`](http://AGENTS.md). Muse glimmer will also need the [Hugging Face MCP](https://huggingface.co/mcp) and/or the [Hugging Face CLI and Skills](https://huggingface.co/docs/hub/en/agents-skills).\n\nInference Endpoint deployment prompt\n\n```text\n## Hugging Face Inference Endpoint deployment\n\nWhen asked to deploy on Hugging Face Inference Endpoints, perform the work; do\nnot give instructions.\n\n1. Inspect Hugging Face authentication, the current model repository, and any\n existing endpoints.\n2. Confirm the exact model repository and immutable revision through the Hub\n API; inspect its architecture, configuration, and chat template.\n3. Confirm that the model is supported by vLLM, then deploy or update a\n protected Inference Endpoint using the managed native vLLM engine.\n4. Choose an available region and the smallest suitable accelerator. Use one\n replica and enable scale-to-zero when supported.\n5. Preserve the previous endpoint configuration for rollback. Do not expose\n tokens, publish private weights, or replace an unrelated endpoint.\n6. Wait for the endpoint to become ready. If startup fails, inspect the logs\n and report the actual blocker rather than repeatedly changing settings.\n7. Validate `/health`, `/v1/models`, and `/v1/chat/completions`, requiring the\n expected model and non-empty, correct content. When agent use is required,\n also validate a real structured tool call.\n8. Configure the Claw agent to use the endpoint's OpenAI-compatible `/v1` URL,\n storing credentials as secrets and retaining the previous provider as\n rollback. Test the connection in a fresh session.\n9. Report concise progress and finish with the repository, revision, engine,\n hardware, endpoint URL, scaling state, and validation results. Claim\n completion only after every required check passes.\n```\n\n### Hey Muse Glimmer, optimize yourself\n\nFinally, let’s get Muse Glimmer to do some light RSI. We can instruct our agent to optimize its own inference engine for specific hardware, in this case a Nvidia H100. To do this, the agent will need to use another inference engine, like Inference Endpoints above.\n\nMuse Glimmer benchmarks its own single-H100 serving stack, testing one reversible change at a time while holding the workload fixed. It keeps only correctness-passing gains and finishes with the fastest reproducible configuration. Here’s the prompt we added to [`AGENTS.md`](http://AGENTS.md). Muse glimmer need the [Hugging Face MCP](https://huggingface.co/mcp) and the [Hugging Face CLI and Skills](https://huggingface.co/docs/hub/en/agents-skills).\n\nSelf-optimization prompt\n\n```text\nYou are Muse Glimmer acting as an autonomous inference-optimization engineer for your own serving stack.\n\nGoal: maximize valid single-H100 aggregate completion throughput in tokens/second.\n\nProtocol:\n1. Establish a correctness-passing baseline.\n2. Test one reversible optimization at a time.\n3. Keep the prompt, concurrency, sampling, request count, warm-up, and decode length fixed.\n4. Reject results that fail correctness or prefix checks.\n5. Record every experiment chronologically with its configuration, raw throughput, correctness, and delta.\n6. Keep improvements and revert regressions.\n7. Stop after six consecutive regressions or when the experiment budget is exhausted.\n8. Report the best valid configuration and exact reproduction command.\n\nCreate a minimal scientific animation of the results:\n- white background;\n- raw tokens/second—never normalize;\n- one point revealed per experiment;\n- connect every point chronologically;\n- begin with the lowest valid result;\n- stop at the best result;\n- export as a GIF.\n\nNever fabricate, interpolate, or count correctness-failing measurements.\n```\n\n[https://huggingface.co/buckets/huggingface/muse-glimmer-assets/resolve/onyx-optimization-progress.gif?download=true](https://huggingface.co/buckets/huggingface/muse-glimmer-assets/resolve/onyx-optimization-progress.gif?download=true)\n\n### Hey Muse Glimmer, research the Hub\n\nTry Muse Glimmer as a Hugging Face research agent. The Gradio Space sends each model request to a private Hugging Face Inference Endpoint through its OpenAI-compatible API. It also connects to the official Hugging Face MCP server, giving the agent read-only tools to search and inspect Hub repositories, models, datasets, Spaces, documentation, and papers.\n\n## Wrapping Up\n\nWe are happy to welcome Muse Glimmer to the Hugging Face Hub. Try [Muse Glimmer](https://huggingface.co/meta-models/Muse-Glimmer-30B) with your local coding setups today!\n", "dateCreated": "2026-10-19T01:26:53.998049939+00:00", "date_created": "2026-08-10T00:00:00+00:00", "date_modified": "2026-08-10T00:00:00+00:00", "date_published": "2026-08-10T00:00:00+00:00", "description": "Great news from the OGs of open source LLMs! Muse Glimmer, released today, is Meta’s new multimodal model, especially designed for local agentic use cases. Distilled from Muse to 30B parameters, and released under the Apache 2.0 license,…", "external_url": "https://huggingface.co/blog/muse-glimmer", "git": { "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e", "path": "items/huggingface-co/2026/08/2026-08-10-meta-is-back-with-muse-glimmer-local-agentic-multimodal-and.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/huggingface-co/2026/08/2026-08-10-meta-is-back-with-muse-glimmer-local-agentic-multimodal-and.md" }, "id": "urn:aggr:item:9f6d421f8ee75e5b19db27f71921447817c72abb", "isBasedOn": { "@id": "https://huggingface.co/blog/muse-glimmer" }, "mainEntity": { "@id": "https://huggingface.co/blog/muse-glimmer", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-08-10-meta-is-back-with-muse-glimmer-local-agentic-multimodal-and/#webpage" }, "dateModified": "2026-08-10T00:00:00+00:00", "datePublished": "2026-08-10T00:00:00+00:00", "headline": "Meta is back with Muse Glimmer: local, agentic, multimodal, and open source", "image": { "@type": "ImageObject", "caption": null, "height": 90, "url": "https://huggingface.co/blog/assets/muse-glimmer/thumbnail.png", "width": 160 }, "timeRequired": "PT16M", "url": "https://huggingface.co/blog/muse-glimmer", "wordCount": 3590 }, "name": "Archived snapshot: Meta is back with Muse Glimmer: local, agentic, multimodal, and open source", "preview": { "alt": null, "color": null, "height": 90, "placeholder": { "data_url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII=", "hash": "" }, "url": "https://huggingface.co/blog/assets/muse-glimmer/thumbnail.png", "width": 160 }, "reading_minutes": 16, "replicated_at": null, "source": "huggingface-co", "source_name": "Hugging Face - Blog", "summary": null, "tags": [], "temporalCoverage": "2026-10-19T01:26:53.998049939+00:00", "title": "Meta is back with Muse Glimmer: local, agentic, multimodal, and open source", "url": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-08-10-meta-is-back-with-muse-glimmer-local-agentic-multimodal-and/", "word_count": 3590 }