.endpoints.huggingface.cloud/v1/chat/completions\" \\\n -H \"Authorization: Bearer $HF_TOKEN\" \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"meta-models/Muse-Spark-Open-816B-A42B-NVFP4\",\n \"messages\": [{\n \"role\": \"user\",\n \"content\": [\n {\"type\": \"image_url\", \"image_url\": {\"url\": \"https://huggingface.co/datasets/merve/vl-test-suite/resolve/main/pills.jpg\"}},\n {\"type\": \"text\", \"text\": \"Do any of the components in this supplement interact?\"}\n ]\n }],\n \"chat_template_kwargs\": {\"enable_thinking\": false}\n }'\n```\n\nAny OpenAI-compatible client works with the same URL and your Hugging Face token as the API key.\n\n## Fine-tuning with TRL\n\nYou can fine-tune the 816B checkpoint with the TRL `SFTTrainer`. This section shows how we fine-tuned it with TRL on 8 nodes of 8 H100 GPUs. The recipe trains LoRA adapters on the attention layers. The experts stay frozen. The full script is in the [TRL examples](https://github.com/huggingface/trl/tree/main/examples/sft_muse_spark_open).\n\nFirst, we need to split the model across 64 GPU devices. Given the large size of the model, we need to parallelize it across multiple axes. We shard the model with FSDP and split the experts across GPUs, using `DistributedConfig`, which tells `transformers` how to split the model when it loads:\n\n```py\nfrom transformers.distributed import DistributedConfig\n\ndistributed_config = DistributedConfig(fsdp_size=64, ep_size=8)\n```\n\nIn the snippet above, `fsdp_size=64` shards the model across all 64 GPUs with FSDP2, and `ep_size=8` splits the 256 experts of each layer into 8 groups. Transformers sends each token to the GPU that holds its expert, so each GPU trains on its own slice of the batch.\n\nNow, we can configure the `SFTConfig` for our training run. We’ve performed a simple task here to adapt the model’s chat template on an open source dataset, but you could apply it to your own use case.\n\n```py\nimport torch\nfrom datasets import load_dataset\nfrom peft import LoraConfig\nfrom transformers.distributed import DistributedConfig\nfrom trl import SFTConfig, SFTTrainer\n\ntrain_dataset = load_dataset(\"allenai/tulu-3-sft-mixture\", split=\"train[:500000]\")\n\ntraining_args = SFTConfig(\n output_dir=\"muse-spark-lora-sft\",\n model_init_kwargs={\"dtype\": torch.bfloat16, \"distributed_config\": DistributedConfig(fsdp_size=64, ep_size=8)},\n per_device_train_batch_size=1,\n max_steps=500,\n max_length=2048,\n packing=True,\n save_only_model=True,\n logging_steps=1,\n report_to=\"trackio\",\n)\n\ntrainer = SFTTrainer(\n model=\"meta-models/Muse-Spark-Open-816B-A42B\",\n args=training_args,\n train_dataset=train_dataset,\n peft_config=LoraConfig(r=16, lora_alpha=32, target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\"]),\n)\ntrainer.train()\n```\n\nThe core parts here are `packing=True` which joins examples into 2048-token sequences without padding. A block-diagonal mask keeps the examples separate. `save_only_model=True` saves the adapter and no optimizer state, because the trainer cannot resume a model that was sharded at load time. `LoraConfig` targets the four attention projections and the 256 experts of each layer stay frozen.\n\nThen we can run this command across each node:\n\n```sh\ntorchrun --nnodes 8 --nproc_per_node 8 --rdzv_backend c10d --rdzv_endpoint \"$MASTER_IP:29700\" sft_muse_spark_open.py\n```\n\nWe trained for 500 steps, which took about 2 hours on 64 H100s, at about 8,000 tokens per second. Mean token accuracy rose from 72% to 84% and the loss fell from 1.26 to 0.57, which signals a good fine-tuning run.\n\n[](https://huggingface.co/datasets/merve/meta-muse-spark-open-assets/resolve/main/training_loss.png)\n\n[](https://huggingface.co/datasets/merve/meta-muse-spark-open-assets/resolve/main/token_acc.png)\n\n## Demos\n\nMuse Spark Open is a coding and agentic model. We wanted to see that for ourselves, so we tried a mixture of multi-modal and agentic tasks that use the model's image and audio inputs. Everything below is something you can run or fork.\n\n### 3D Asset Generator\n\nWe combined 2-bit quantized Muse Spark open (served on llama.cpp) with the Pi coding agent to generate and iteratively improve 3D assets in [Three.js](http://Three.js). You can find the prompt [here](https://huggingface.co/datasets/merve/meta-muse-spark-open-assets/resolve/main/pi-3d-prompt.txt), as the model is quantized it requires a bit more steering. We get 18 TPS on llama.cpp on Mac Ultra.\n\n[](https://huggingface.co/datasets/merve/meta-muse-spark-open-assets/resolve/main/pi-demo.png)\n\nWe initially describe the asset, and Pi uses the model to generate Three.js code for the asset, renders the result in a browser, and captures screenshots from multiple angles. It then sends those images back to the same model on llama server for visual feedback. Pi uses that feedback to edit the code and repeats the process.\n\n### Browser Interactions with OpenEnv and Hugging Face Sandbox\n\nWe tested Spark on computer use tasks and were amazed by the performance, so we have built a lean demo that also serves to teach you how a computer use agent works. You can also further customize this for your needs. This is both an educational example and something you can run in a few lines. You can find it [here](https://huggingface.co/docs/openenv/main/en/environments/helium_browser).\n\nMuse Spark NVFP4 Browser Agent with OpenEnv and HF Sandboxes\n\n[Hugging Face Sandboxes](https://huggingface.co/docs/huggingface_hub/en/guides/sandbox) are sandboxes built on Hugging Face Jobs for easy code execution. The demo consists of three components:\n\n1. Muse Spark Open served with Inference Endpoints,\n2. OpenEnv wrapping web driver actions with helium and selenium (click, type, scroll etc) defining browser environment’s reset, step, observation, termination interface,\n3. Chromium web browser\n\nwhere the remote sandbox contains the environment and the browser as illustrated below.\n\n[](https://huggingface.co/datasets/merve/meta-muse-spark-open-assets/resolve/main/browser_server.png)\n\nYou can easily try this with a few steps, described in the example. Start Inference Endpoint with Muse Spark Open (NVFP4 works well too!), add your endpoint URL as environmental variable, publish your OpenEnv server in Spaces and also provide that, then you can run the agent loop. Muse Spark Open sees the previous actions, system prompt, url and the screenshot of the browser to take an action, and nothing else (no DOM, no HTML code, nothing) so it’s a truly multimodal browser demo.\n\n### Call your Browser\n\nFinally, we wanted to push Spark’s agentic multimodality to the limits, so we set up [this space](https://huggingface.co/spaces/burtenshaw/muse-spark-call-your-browser) that combines the model’s audio, image, and agentic properties into a browser agent. We give the model a general description of a thing we need and it interacts with the integrated browser to find options on Amazon.\n\nMuse Spark Open takes the voice input and reasons over the type of product that meets the descriptions, searches for it on Amazon, and selects the most appropriate based on criteria.\n\n## Wrapping Up\n\nMuse Spark Open is a full-size open-weight model from the Muse series. Start with the [BF16](https://huggingface.co/meta-models/Muse-Spark-Open-816B-A42B) or [NVFP4](https://huggingface.co/meta-models/Muse-Spark-Open-816B-A42B-NVFP4) checkpoint, or try one of the great `llama.cpp quants`. If you build something with Muse Spark Open, share it, and tell us in the [repository’s discussions.](https://huggingface.co/meta-models/Muse-Spark-Open-816B-A42B/discussions)\n",
"authors": [],
"captured_at": "2026-10-15T18:36:21.248140220+00:00",
"category": "ai",
"content_html": "Muse Spark Open is an open-weight model from Meta. The mixture of experts model brings text, images, audio, and video into a model with approximately 816 billion parameters, around 42 billion active per token, and a context window of up to one million tokens. It generates text, giving developers a common interface for asking questions about different kinds of input.
\nFollowing Muse Glimmer, Muse Spark Open introduces a mixture-of-experts architecture at a much larger scale, and the most capable US open-weights model to date.
\nMeta trained Muse Spark Open for agentic coding: reading a codebase, writing and debugging code, and calling tools. It also takes images, audio, and video as input, so the same model can read a screenshot, listen to a voice note, or watch a screen recording while it works on your code. This post shows you how to run it, how to fine-tune it, and what we built with it. Find all the models here.
\nOverall Capabilities and Architecture
\nMeta describes Muse Spark Open as a coding-optimized model trained on real developer workflows such as multi-file refactors and long debugging sessions. Muse Spark Open is an agentic focused model that plans, acts, observes, and self-corrects over long tasks, including browsing the web with images.
\n\n\n\n| Feature | \nMuse Spark Open | \n
\n\n\n\n| Model size | \n~816B total parameters; ~42B active per token | \n
\n\n| Expert routing | \n256 experts per MoE layer; 8 selected per token | \n
\n\n| Inputs | \nText, images, audio, and video | \n
\n\n| Output | \nText | \n
\n\n| Context window | \nUp to 1M tokens | \n
\n\n| Checkpoint formats | \nBF16 and NVFP4 | \n
\n\n
\nThe checkpoint repositories are Muse-Spark-Open-816B-A42B and Muse-Spark-Open-816B-A42B-NVFP4.
\n![\"model](\"https://huggingface.co/datasets/merve/meta-muse-spark-open-assets/resolve/main/model.png\")
\nMuse Spark Open is the full-size model of the Muse series of models. Muse Glimmer, which Meta released in August, is a small dense model from the same series, ideal for running locally. Both models use the same tokenizer and vocabulary. The chat template follows the Muse Glimmer template, with additions for the new modalities. In Muse Spark Open the input embedding and the output layer are separate matrices of about 1.66B parameters each.
\nMixture of Experts
\nThe language model has 62 layers and a hidden dimension of 8192. Each layer has 128 query heads and 16 key-value heads (grouped-query attention). Each head has 64 dimensions. The attention block has an output gate and an output normalization layer. The model also applies normalization to the queries and the keys (QK-norm). Each block combines its input and its output through a learned residual gate, not through a plain sum.
\nMuse Glimmer uses one dense feed-forward network in each layer. Muse Spark Open replaces this dense network with a Mixture-of-Experts (MoE) block. Each MoE block has 256 experts, and a router selects 8 experts for each token. Each expert is a feed-forward network that projects from 8192 to 4096 dimensions and back to 8192, then applies its own normalization to its output. The experts hold ~97% of the total model parameters.
\nOnly 8 experts run per token, so the model computes each token with about 42B parameters. All 256 experts stay in memory, so deployment needs the memory of an 816B-parameter model, but runs at the much faster speed of a 42B model. The BF16 checkpoint is about 1.5 TB in safetensors format. The NVFP4 checkpoint stores the parameters in 4-bit precision with negligible loss in accuracy, specifically for NVIDIA Blackwell GPUs. In transformers, the BF16 checkpoint runs across 4 nodes of 8 H200 GPUs with tensor parallelism (TP=32) and the NVFP4 runs on a single B200 node. If you want to deploy Muse Spark Open, check out the llama.cpp, vLLM, and SGLang recipes below.
\nPerception Encoder
\nA perception encoder converts images and video frames into tokens for the language model, just like Muse Glimmer. The encoder has 1.8B parameters, 50 layers, a hidden dimension of 1536, and 16 attention heads. It splits each image into 14×14 patches and produces features with 4096 dimensions. Position information comes from learned absolute embeddings and 2D RoPE. The feed-forward network in each layer uses LayerNorm and GELU. Every fourth layer and the last layer use full attention, and the other layers use sparse attention.
\n![\"image](\"https://huggingface.co/datasets/merve/meta-muse-spark-open-assets/resolve/main/image_pipeline.png\")
\nMuse Glimmer 30B uses an encoder with the same architecture. The design follows the Perception Encoder work from Meta. Video input goes through the same encoder as images.
\nAudio Encoder
\n![\"audio](\"https://huggingface.co/datasets/merve/meta-muse-spark-open-assets/resolve/main/audio_encoder.png\")
\nMuse Spark Open accepts audio as input through its audio encoder. An audio tokenizer converts the audio into tokens, and the chat template marks the audio position with <|audio|> tokens. The encoder emits one audio token every 80 ms, which yields 12.5 tokens per second. So let’s say, you have an audio of 10 seconds, the number of input audio tokens would be 125. The encoder itself is a convolutional pyramid where each stage consists of a patch embedder of different patch sizes and 2 convolutional blocks of 1d kernels of size 7 with stride of 1. These stages are repeated for patch sizes of [16, 6, 5, 4]. At the end of the final stage, the temporal frames are fed to a final block of RMSNorm → linear projection → RMSNorm.
\nLong Context
\nThe context window is 1M tokens at inference. The configuration sets max_position_embeddings to 1,048,576. This includes image, video, and audio tokens in the same window as text.
\nGrouped-query attention keeps the KV cache small for a model of this size. Each token adds 2 × 16 heads × 64 dimensions × 62 layers = 126,976 values to the cache. In BF16, this is about 250 KB per token and about 266 GB for a full 1M-token context.
\nInference Support
\nTransformers
\nThe Muse Spark Open implementation provides MuseSparkForConditionalGeneration for text generation from multimodal inputs and MuseSparkProcessor for preparing conversations. The processor combines the tokenizer, image and video processors, audio feature extractor, and chat template.
\nUse a DistributedConfig to configure the parallelism settings. tp_size is the number of devices across which the dense layers are sharded, and ep_size is the number of devices across which the experts are distributed.
\nYou can initialize the model and processor as follows.
\nimport os\n\nfrom transformers import AutoModelForMultimodalLM, AutoTokenizer, DistributedConfig\n\ndistributed_config = DistributedConfig(\n tp_size=int(os.environ["WORLD_SIZE"]),\n ep_size=int(os.environ["WORLD_SIZE"]),\n)\n\ntokenizer = AutoTokenizer.from_pretrained("meta-models/Muse-Spark-Open-816B-A42B\n")\nmodel = AutoModelForMultimodalLM.from_pretrained(\n "meta-models/Muse-Spark-Open-816B-A42B",\n distributed_config=distributed_config,\n)\n\nmessages = [\n {"role": "user", "content": "Explain in one sentence why the sky is usually blue."},\n]\ninputs = tokenizer.apply_chat_template(\n messages,\n reasoning_effort="medium", add_generation_prompt=True,\n tokenize=True,\n return_dict=True,\n return_tensors="pt",\n).to(model.device).to(model.dtype)\n\ninput_len = inputs["input_ids"].shape[-1]\n\noutputs = model.generate(**inputs, max_new_tokens=2000)\n
\nText-only Inference
\nPrepare the conversation, then pass the messages to processor.apply_chat_template, enabling add_generation_prompt=True, tokenize=True, return_dict=True, and return_tensors="pt". Pass the processor output to model.generate and decode the newly generated tokens, excluding the input prefix. The processor has a lever for the reasoning effort, mapping symbolic levels (low/medium/high/xhigh) to the integers the model itself accepts (16, 64, 256, 512). Note that turning off reasoning is not possible with Muse Spark Open, so keep the model reasoning at “low” if you want to consume less tokens to solve an easier task. You can also set a numeric effort by including a line like Reasoning strength: 1 in the system prompt.
\nmessages = [\n {"role": "system", "content": "You should only answer with a number."},\n {"role": "user", "content": "What is 17 * 23?"},\n]\n\ninputs = processor.apply_chat_template(\n messages,\n add_generation_prompt=True,\n tokenize=True,\n return_dict=True,\n return_tensors="pt",\n reasoning_effort="high", ).to(model.device).to(model.dtype)\n\noutput = model.generate(**inputs, max_new_tokens=2000)\ngenerated_tokens = output[0][inputs["input_ids"].shape[1] :]\nprint(processor.decode(generated_tokens, skip_special_tokens=False))\n\n
\nImages and Text
\nThe Muse Spark Open processor expands each image marker into the number of patch tokens required by the processed image, surrounded by image boundary tokens. This expansion keeps the text positions aligned with the visual features passed to the model. The processor also handles multiple images within a single prompt.
\nimage_url = (\n "https://huggingface.co/datasets/merve/vl-test-suite/"\n "resolve/main/pills.jpg"\n)\nmessages = [\n {\n "role": "user",\n "content": [\n {\n "type": "image",\n "image": image_url,\n },\n {\n "type": "text",\n "text": "Do any of the components in this supplement interact?",\n },\n ],\n },\n]\n\ninputs = processor.apply_chat_template(\n messages,\n tokenize=True,\n add_generation_prompt=True,\n reasoning_effort="medium",\n return_dict=True,\n return_tensors="pt",\n).to(model.device).to(model.dtype)\ninput_len = inputs["input_ids"].shape[-1]\n\noutputs = model.generate(**inputs, max_new_tokens=2000)\nresponse = processor.decode(outputs[0][input_len:], skip_special_tokens=False)\n\nprocessor.parse_response(response)\n
\nVideo Inference
\nVideo prompts include frame timing as well as visual content. The processor handles videos with audios in them (which is an upgrade from Muse Glimmer thanks to the audio encoder). The processor downsamples video frames and audio to align frames together. By default, videos are sampled to 2 FPS, capped at 96 frames. You can modify this by setting the fps of the video processor:
\nprocessor.video_processor.fps = 4.0\n
\nAs a general rule of thumb, videos are frames in temporal sequence, and the more you downsample, the more you will lose frames in between specific events, so you need to keep fps high if you want more accurate generations. However, there’s a trade-off because you will require more memory and increase latency for prefill. The frames are interleaved with text input that states timestamps (e.g. “Time: 3.5s”). To enable passing in videos with audio, pass load_audio_from_video=True to apply_chat_template.
\nPreserve the video's metadata when preparing inputs with return_metadata=True so the processor can assign the correct framerate.
\nimport torch\n\nmessages = [\n {\n "role": "user",\n "content": [\n {\n "type": "text",\n "text": (\n "Describe what happens in this video. "\n "Use timestamps and include relevant speech or sounds."\n ),\n },\n {\n "type": "video",\n "url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/ramen.mp4"\n },\n ],\n }\n]\n\ninputs = processor.apply_chat_template(\n messages,\n add_generation_prompt=True,\n tokenize=True,\n return_dict=True,\n return_tensors="pt",\n load_audio_from_video=True,\n return_metadata=True,\n).to(model.device).to(model.dtype)\n\nwith torch.inference_mode():\n output_ids = model.generate(\n **inputs,\n max_new_tokens=256,\n do_sample=False,\n )\n\ngenerated_ids = output_ids[:, inputs["input_ids"].shape[1] :]\nresponse = processor.batch_decode(\n generated_ids,\n skip_special_tokens=True,\n)[0]\n\nprint(response)\n
\nAudio Inference
\nThe Muse Spark Open audio tower consumes a raw waveform. The processor expects mono audio at 24 kHz and pads the waveform to a multiple of 1,920 samples, corresponding to one audio embedding per 80 milliseconds. Audio supplied directly to the processor at another sample rate must first be resampled. You can do this by using the load_audio util of transformers. load_audio uses torchcodec, librosa or torchaudio backends, so make sure to have one of these installed locally before running inference.
\nIf you want to simply use apply_chat_template with messages though, this is all handled for you under the hood:
\nimport torch\nfrom transformers import AutoProcessor, AutoModelForMultimodalLM\n\nmodel_id = "meta-models/Muse-Spark-Open-816B-A42B"\nprocessor = AutoProcessor.from_pretrained(model_id)\n\nmodel = AutoModelForMultimodalLM.from_pretrained(model_id).eval()\n\nmessages = [\n {\n "role": "user",\n "content": [\n {\n "type": "audio",\n "url": "https://huggingface.co/buckets/huggingface/audio-samples/resolve/bbaudio_question_39.mp3",\n },\n {\n "type": "text",\n "text": "Answer the question.",\n },\n ],\n }\n]\n\ninputs = processor.apply_chat_template(\n messages,\n add_generation_prompt=True,\n tokenize=True,\n return_dict=True,\n return_tensors="pt",\n load_audio_backend="auto",\n).to(model.device).to(model.dtype)\n\nwith torch.inference_mode():\n output_ids = model.generate(\n **inputs,\n max_new_tokens=128,\n do_sample=False,\n )\n\nprompt_length = inputs["input_ids"].shape[1]\ngenerated_text = processor.batch_decode(\n output_ids[:, prompt_length:],\n skip_special_tokens=True,\n)\n\nprint(generated_text[0])\n
\nThe processor inserts the matching number of audio tokens between audio boundary markers. It also returns a mask identifying real samples, allowing the model to exclude padded audio embeddings.
\nTool Calling
\nMuse Spark Open can do text-only and multimodal tool calling. You can make the model call tools as follows.
\n\nfrom transformers import AutoProcessor, AutoModelForMultimodalLM\n\nMODEL_ID = "meta-models/Muse-Spark-Open-816B-A42B"\n\nprocessor = AutoProcessor.from_pretrained(MODEL_ID)\nmodel = AutoModelForMultimodalLM.from_pretrained(\n MODEL_ID,\n dtype="auto",\n device_map="auto"\n)\n\ntools = [\n {\n "type": "function",\n "function": {\n "name": "weather.get",\n "description": "Get the current weather for a city.",\n "parameters": {\n "type": "object",\n "properties": {\n "city": {"type": "string"},\n },\n "required": ["city"],\n },\n },\n }\n]\n\n\nmessages = [\n {\n "role": "user",\n "content": [\n {"type": "image", "image": "https://huggingface.co/datasets/merve/vl-test-suite/resolve/main/SF.png"},\n {"type": "text", "text": "I'm going to the city in this picture. What clothes should I wear?"},\n ],\n },\n]\n\ninputs = processor.apply_chat_template(\n messages,\n tools=tools,\n tokenize=True,\n return_dict=True,\n return_tensors="pt",\n add_generation_prompt=True,\n reasoning_strength="high"\n).to(model.device).to(model.dtype)\n\ninput_len = inputs["input_ids"].shape[-1]\noutputs = model.generate(**inputs)\nresponse = processor.parse_response(\n completion,\n prefix=inputs["input_ids"][0],\n tools=tools,\n )\nprint(response)\n
\nMuse Spark Open uses a channel-based chat template (Harmony, first introduced by gpt-oss). Note that you don’t have to know about the internals, but it’s useful to check the channels the model routes to, in case you get low quality outputs. In the input message below:
\n<|begin_of_text|>\n <|start|>system<|message|>\n You are a helpful AI assistant.\n Knowledge cutoff: 2026-01-04.\n Current date: ...\n ...\n <|eot|>\n <|start|>user<|message|>...\n <|eot|>\n <|start|>assistant\n
\n<|start|>assistant is intentionally left open, it’s the model that chooses recipient with to= . The model has different recipients, e.g. self asks the model to reason,
\nto=self<|message|>the model reasons here<|eom|> # reasoning\n<|start|>assistant to=user<|message|>final answer<|eot|> # final answer\n
\nself is for reasoning, commentary are comments before and between tool calls visible to the end user, user is the final response.
\nTool-call routing is parsed as follows (assuming you have passed tools to apply_chat_template:
\n<|start|>assistant to=weather.get<|message|>\n <atem:function_calls>\n <atem:invoke name="weather.get">\n <atem:parameter name="city">San Francisco</atem:parameter>\n </atem:invoke>\n </atem:function_calls>\n <|eot|>\n
\nLlama.cpp
\nMuse Spark 2-bit running on Llama Server
\nMuse Spark Open is integrated in llama.cpp and comes in the prebuilt binaries. You can install this with following command:
\ncurl -LsSf https://llama.app/install.sh | sh\n
\nYou can start the server as follows with quant suffix attached to model ID:
\n\nllama serve -hf ggml-org/Muse-Spark-Open-816B-A42B-GGUF:Q2_K\n
\nThe repository containing quants is here. Once the server has started, you can head to localhost:9931 to chat with the built-in WebUI. From the web UI you can add audio and images for inference and adjust the reasoning lever.
\nNote that turning off reasoning is not possible with Muse Spark Open, so keep the model reasoning at “low” if you want to consume less tokens to solve an easier task.
\nWe tested quants on Mac M3 Ultra (512GB). An imatrix-calibrated 2-bit quant runs at 18 tokens/s, with plenty of room free for the KV cache (304 GB used by the model).
\nYou can also query the server as follows. Feel free to check out the API reference for multimodality, tool calls and more.
\ncurl http://localhost:8080/v1/chat/completions \\\n -H "Content-Type: application/json" \\\n -d '{\n "messages": [\n {"role": "system", "content": "You are a helpful assistant."},\n {"role": "user", "content": "Write a limerick about python exceptions"}\n ]\n }'\n\ncurl -s http://127.0.0.1:8080/v1/chat/completions \\\n -H "Content-Type: application/json" -H "Authorization: Bearer <your-key>" \\\n -d '{"model":"muse-spark-open","messages":[{"role":"user","content":[\n {"type":"text","text":"What is in this image?"},\n {"type":"image_url","image_url":{"url":"data:image/png;base64,..."}}\n ]}],"temperature":0}'\n\ncurl -s http://127.0.0.1:8080/v1/chat/completions \\\n -H "Content-Type: application/json" -H "Authorization: Bearer <your-key>" \\\n -d '{"model":"muse-spark-open","messages":[{"role":"user","content":[\n {"type":"text","text":"Transcribe this audio."},\n {"type":"input_audio","input_audio":{"data":"data:audio/wav;base64,..."}}\n ]}],"temperature":0}'\n
\nllama.cpp team ships different quantized versions. We have tested Q2_K, Q4_K_S and Q_8 on Apple Silicon chips. You can run each quant by adding the quant id as suffix, llama serve -hf ggml-org/Muse-Spark-Open-816B-A42B:Q2_K.
\nSGLang
\nThe Muse Spark Open integration for SGLang supports text, image, audio, and video inputs. The NVFP4 reference configuration runs across four GB300 GPUs.
\nFrom the SGLang integration environment, download the checkpoint from the Hugging Face Hub and start the server:
\nexport MODEL_ID=meta-models/Muse-Spark-Open-816B-A42B-NVFP4\nexport CKPT="$(hf download "$MODEL_ID")"\n\nSGLANG_FLASHINFER_NVFP4_PER_TOKEN_ACTIVATION=1 \\\nuv run sglang_meta serve \\\n --model-path "$CKPT" \\\n --served-model-name "$MODEL_ID" \\\n --quantization modelopt_fp4 \\\n --moe-runner-backend flashinfer_trtllm \\\n --tp-size 4 \\\n --ep-size 4 \\\n --port 30000\n
\nThe checkpoint includes the chat template, which we pass explicitly. The launcher configures the Muse reasoning and tool-call parsers.
\nSend a request through the OpenAI-compatible API:
\ncurl http://localhost:30000/v1/chat/completions \\\n -H "Content-Type: application/json" \\\n -d '{\n "model": "meta-models/Muse-Spark-Open-816B-A42B-NVFP4",\n "messages": [{\n "role": "user",\n "content": "Explain how a mixture-of-experts model works."\n }],\n "max_tokens": 2048\n }'\n
\nThe response separates the reasoning trace in reasoning_content from the final answer in content.
\nvLLM
\nThe Muse Spark Open reference integration for vLLM supports text and image inputs with the NVFP4 checkpoint that require blackwell series GPUs.
\nAfter installing the reference plugin and its dependencies, download the checkpoint from the Hugging Face Hub and launch the server:
\nexport MODEL_ID=meta-models/Muse-Spark-Open-816B-A42B-NVFP4\nexport CKPT="$(hf download "$MODEL_ID")"\n\nVLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \\\nVLLM_PLUGINS=musespark1x_models,ptq_nvfp4 \\\nvllm serve "$CKPT" \\\n --served-model-name "$MODEL_ID" \\\n --tokenizer-mode musespark1x \\\n --config-format auto \\\n --load-format internal_safetensor \\\n --tensor-parallel-size 4 \\\n --enable-expert-parallel \\\n --max-model-len 131072 \\\n --gpu-memory-utilization 0.92 \\\n --reasoning-parser musespark1x \\\n --tool-call-parser musespark1x \\\n --enable-auto-tool-choice \\\n --port 8000\n
\nThis configuration sets a 128K-token serving context. The plugin supplies the model implementation, checkpoint loader, and reasoning and tool-call parsers.
\nSend a request using the same Hub model ID:
\ncurl http://localhost:8000/v1/chat/completions \\\n -H "Content-Type: application/json" \\\n -d '{\n "model": "meta-models/Muse-Spark-Open-816B-A42B-NVFP4",\n "messages": [{\n "role": "user",\n "content": "Explain how a mixture-of-experts model works."\n }],\n "max_tokens": 2048\n }'\n
\nMLX
\nMuse Spark Open is available in oMLX and mlx-lm today. oQ quants optimized for oMLX are available in 3-bit and 4-bit, ready to use right away.
\nWe tested the oQ3e quant on an M3 Ultra (512 GB). The model takes about 344 GB, leaving room for the KV cache, and runs at about 225 tok/s prefill and 21 tok/s generation with a 1024-token prompt and 128 generated tokens.
\nOnce you download the model through the oMLX interface, you can chat with it in the built-in chat UI. From the chat UI you can add images and adjust the reasoning effort on the fly, from low to xhigh. The model’s reasoning shows up as a collapsible thinking block, separate from the final answer.
\nNote that it's not possible to turn reasoning off, and the default (Auto) uses xhigh. Set it to “low” if you want to use fewer tokens on easier tasks. ![\"omlx\"](\"https://huggingface.co/datasets/merve/meta-muse-spark-open-assets/resolve/main/omlx_muse_spark_open_reasoning_screenshot.png\")
\nThe language model is also available in this PR of the official mlx-lm repository. You can run it on the same high-quality oQ3e quant like this:
\nmlx_lm.generate --model mlx-community/Muse-Spark-Open-816B-A42B-oQ3e \\\n --prompt "Explain convolutional networks like I am 5" \\\n --max-tokens 5000\n
\nThis is a big model, even the 3-bit quantization requires more than 256 GB of RAM, as explained above. You can also run distributed inference across multiple computers, please refer to the documentation for details.
\nDeploying Muse Spark Open with Inference Endpoints
\nYou can deploy Muse Spark Open on Inference Endpoints with a pre-validated SGLang recipe for the NVFP4 variant which runs on either 8 × NVIDIA RTX Pro 6000 98 GB with 131072 context length, or with 8 x NVIDIA H200 141GB if you want 1048576 context length instead.
\nTo deploy, run one command with the Hugging Face CLI:
\nhf endpoints catalog deploy --repo meta-models/Muse-Spark-Open-816B-A42B-NVFP4 --name muse-spark\n
\nOr open endpoints.huggingface.co/new/meta-models/Muse-Spark-Open-816B-A42B-NVFP4 and select Create Endpoint.
\nThe endpoint exposes an OpenAI-compatible API with both Chat Completions API at /v1/chat/completions and Responses API at /v1/responses.
\ncurl "https://<your-endpoint>.endpoints.huggingface.cloud/v1/chat/completions" \\\n -H "Authorization: Bearer $HF_TOKEN" \\\n -H "Content-Type: application/json" \\\n -d '{\n "model": "meta-models/Muse-Spark-Open-816B-A42B-NVFP4",\n "messages": [{\n "role": "user",\n "content": [\n {"type": "image_url", "image_url": {"url": "https://huggingface.co/datasets/merve/vl-test-suite/resolve/main/pills.jpg"}},\n {"type": "text", "text": "Do any of the components in this supplement interact?"}\n ]\n }],\n "chat_template_kwargs": {"enable_thinking": false}\n }'\n
\nAny OpenAI-compatible client works with the same URL and your Hugging Face token as the API key.
\nFine-tuning with TRL
\nYou can fine-tune the 816B checkpoint with the TRL SFTTrainer. This section shows how we fine-tuned it with TRL on 8 nodes of 8 H100 GPUs. The recipe trains LoRA adapters on the attention layers. The experts stay frozen. The full script is in the TRL examples.
\nFirst, we need to split the model across 64 GPU devices. Given the large size of the model, we need to parallelize it across multiple axes. We shard the model with FSDP and split the experts across GPUs, using DistributedConfig, which tells transformers how to split the model when it loads:
\nfrom transformers.distributed import DistributedConfig\n\ndistributed_config = DistributedConfig(fsdp_size=64, ep_size=8)\n
\nIn the snippet above, fsdp_size=64 shards the model across all 64 GPUs with FSDP2, and ep_size=8 splits the 256 experts of each layer into 8 groups. Transformers sends each token to the GPU that holds its expert, so each GPU trains on its own slice of the batch.
\nNow, we can configure the SFTConfig for our training run. We’ve performed a simple task here to adapt the model’s chat template on an open source dataset, but you could apply it to your own use case.
\nimport torch\nfrom datasets import load_dataset\nfrom peft import LoraConfig\nfrom transformers.distributed import DistributedConfig\nfrom trl import SFTConfig, SFTTrainer\n\ntrain_dataset = load_dataset("allenai/tulu-3-sft-mixture", split="train[:500000]")\n\ntraining_args = SFTConfig(\n output_dir="muse-spark-lora-sft",\n model_init_kwargs={"dtype": torch.bfloat16, "distributed_config": DistributedConfig(fsdp_size=64, ep_size=8)},\n per_device_train_batch_size=1,\n max_steps=500,\n max_length=2048,\n packing=True,\n save_only_model=True,\n logging_steps=1,\n report_to="trackio",\n)\n\ntrainer = SFTTrainer(\n model="meta-models/Muse-Spark-Open-816B-A42B",\n args=training_args,\n train_dataset=train_dataset,\n peft_config=LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj", "k_proj", "v_proj", "o_proj"]),\n)\ntrainer.train()\n
\nThe core parts here are packing=True which joins examples into 2048-token sequences without padding. A block-diagonal mask keeps the examples separate. save_only_model=True saves the adapter and no optimizer state, because the trainer cannot resume a model that was sharded at load time. LoraConfig targets the four attention projections and the 256 experts of each layer stay frozen.
\nThen we can run this command across each node:
\ntorchrun --nnodes 8 --nproc_per_node 8 --rdzv_backend c10d --rdzv_endpoint "$MASTER_IP:29700" sft_muse_spark_open.py\n
\nWe trained for 500 steps, which took about 2 hours on 64 H100s, at about 8,000 tokens per second. Mean token accuracy rose from 72% to 84% and the loss fell from 1.26 to 0.57, which signals a good fine-tuning run.
\n![\"Loss\"](\"https://huggingface.co/datasets/merve/meta-muse-spark-open-assets/resolve/main/training_loss.png\")
\n![\"Acc\"](\"https://huggingface.co/datasets/merve/meta-muse-spark-open-assets/resolve/main/token_acc.png\")
\nDemos
\nMuse Spark Open is a coding and agentic model. We wanted to see that for ourselves, so we tried a mixture of multi-modal and agentic tasks that use the model's image and audio inputs. Everything below is something you can run or fork.
\n3D Asset Generator
\nWe combined 2-bit quantized Muse Spark open (served on llama.cpp) with the Pi coding agent to generate and iteratively improve 3D assets in Three.js. You can find the prompt here, as the model is quantized it requires a bit more steering. We get 18 TPS on llama.cpp on Mac Ultra.
\n![\"Pi](\"https://huggingface.co/datasets/merve/meta-muse-spark-open-assets/resolve/main/pi-demo.png\")
\nWe initially describe the asset, and Pi uses the model to generate Three.js code for the asset, renders the result in a browser, and captures screenshots from multiple angles. It then sends those images back to the same model on llama server for visual feedback. Pi uses that feedback to edit the code and repeats the process.
\nBrowser Interactions with OpenEnv and Hugging Face Sandbox
\nWe tested Spark on computer use tasks and were amazed by the performance, so we have built a lean demo that also serves to teach you how a computer use agent works. You can also further customize this for your needs. This is both an educational example and something you can run in a few lines. You can find it here.
\nMuse Spark NVFP4 Browser Agent with OpenEnv and HF Sandboxes
\nHugging Face Sandboxes are sandboxes built on Hugging Face Jobs for easy code execution. The demo consists of three components:
\n\n- Muse Spark Open served with Inference Endpoints,
\n- OpenEnv wrapping web driver actions with helium and selenium (click, type, scroll etc) defining browser environment’s reset, step, observation, termination interface,
\n- Chromium web browser
\n
\nwhere the remote sandbox contains the environment and the browser as illustrated below.
\n![\"Diagram\"](\"https://huggingface.co/datasets/merve/meta-muse-spark-open-assets/resolve/main/browser_server.png\")
\nYou can easily try this with a few steps, described in the example. Start Inference Endpoint with Muse Spark Open (NVFP4 works well too!), add your endpoint URL as environmental variable, publish your OpenEnv server in Spaces and also provide that, then you can run the agent loop. Muse Spark Open sees the previous actions, system prompt, url and the screenshot of the browser to take an action, and nothing else (no DOM, no HTML code, nothing) so it’s a truly multimodal browser demo.
\nCall your Browser
\nFinally, we wanted to push Spark’s agentic multimodality to the limits, so we set up this space that combines the model’s audio, image, and agentic properties into a browser agent. We give the model a general description of a thing we need and it interacts with the integrated browser to find options on Amazon.
\nMuse Spark Open takes the voice input and reasons over the type of product that meets the descriptions, searches for it on Amazon, and selects the most appropriate based on criteria.
\nWrapping Up
\nMuse Spark Open is a full-size open-weight model from the Muse series. Start with the BF16 or NVFP4 checkpoint, or try one of the great llama.cpp quants. If you build something with Muse Spark Open, share it, and tell us in the repository’s discussions.
\n",
"content_markdown": "Muse Spark Open is an open-weight model from Meta. The mixture of experts model brings text, images, audio, and video into a model with approximately 816 billion parameters, around 42 billion active per token, and a context window of up to one million tokens. It generates text, giving developers a common interface for asking questions about different kinds of input.\n\nFollowing Muse Glimmer, Muse Spark Open introduces a mixture-of-experts architecture at a much larger scale, and the most capable US open-weights model to date.\n\nMeta trained Muse Spark Open for agentic coding: reading a codebase, writing and debugging code, and calling tools. It also takes images, audio, and video as input, so the same model can read a screenshot, listen to a voice note, or watch a screen recording while it works on your code. This post shows you how to run it, how to fine-tune it, and what we built with it. Find all the models [here](https://huggingface.co/collections/meta-models/muse-spark-open).\n\n## Overall Capabilities and Architecture\n\nMeta describes Muse Spark Open as a coding-optimized model trained on real developer workflows such as multi-file refactors and long debugging sessions. Muse Spark Open is an agentic focused model that plans, acts, observes, and self-corrects over long tasks, including browsing the web with images.\n\n| Feature | Muse Spark Open |\n| ------------------ | ----------------------------------------------- |\n| Model size | \\~816B total parameters; ~42B active per token |\n| Expert routing | 256 experts per MoE layer; 8 selected per token |\n| Inputs | Text, images, audio, and video |\n| Output | Text |\n| Context window | Up to 1M tokens |\n| Checkpoint formats | BF16 and NVFP4 |\n\nThe checkpoint repositories are Muse-Spark-Open-816B-A42B and Muse-Spark-Open-816B-A42B-NVFP4.\n\n[](https://huggingface.co/datasets/merve/meta-muse-spark-open-assets/resolve/main/model.png)\n\nMuse Spark Open is the full-size model of the Muse series of models. Muse Glimmer, which Meta released in August, is a small dense model from the same series, ideal for running locally. Both models use the same tokenizer and vocabulary. The chat template follows the Muse Glimmer template, with additions for the new modalities. In Muse Spark Open the input embedding and the output layer are separate matrices of about 1.66B parameters each.\n\n### Mixture of Experts\n\nThe language model has 62 layers and a hidden dimension of 8192. Each layer has 128 query heads and 16 key-value heads (grouped-query attention). Each head has 64 dimensions. The attention block has an output gate and an output normalization layer. The model also applies normalization to the queries and the keys (QK-norm). Each block combines its input and its output through a learned residual gate, not through a plain sum.\n\nMuse Glimmer uses one dense feed-forward network in each layer. Muse Spark Open replaces this dense network with a Mixture-of-Experts (MoE) block. Each MoE block has 256 experts, and a router selects 8 experts for each token. Each expert is a feed-forward network that projects from 8192 to 4096 dimensions and back to 8192, then applies its own normalization to its output. The experts hold ~97% of the total model parameters.\n\nOnly 8 experts run per token, so the model computes each token with about 42B parameters. All 256 experts stay in memory, so deployment needs the memory of an 816B-parameter model, but runs at the much faster speed of a 42B model. The BF16 checkpoint is about 1.5 TB in safetensors format. The NVFP4 checkpoint stores the parameters in 4-bit precision with negligible loss in accuracy, specifically for NVIDIA Blackwell GPUs. In transformers, the BF16 checkpoint runs across 4 nodes of 8 H200 GPUs with tensor parallelism (TP=32) and the NVFP4 runs on a single B200 node. If you want to deploy Muse Spark Open, check out the `llama.cpp`, vLLM, and SGLang recipes below.\n\n### Perception Encoder\n\nA perception encoder converts images and video frames into tokens for the language model, just like Muse Glimmer. The encoder has 1.8B parameters, 50 layers, a hidden dimension of 1536, and 16 attention heads. It splits each image into 14×14 patches and produces features with 4096 dimensions. Position information comes from learned absolute embeddings and 2D RoPE. The feed-forward network in each layer uses LayerNorm and GELU. Every fourth layer and the last layer use full attention, and the other layers use sparse attention.\n\n[](https://huggingface.co/datasets/merve/meta-muse-spark-open-assets/resolve/main/image_pipeline.png)\n\nMuse Glimmer 30B uses an encoder with the same architecture. The design follows the [Perception Encoder](https://huggingface.co/papers/2504.13181) work from Meta. Video input goes through the same encoder as images.\n\n### Audio Encoder\n\n[](https://huggingface.co/datasets/merve/meta-muse-spark-open-assets/resolve/main/audio_encoder.png)\n\nMuse Spark Open accepts audio as input through its audio encoder. An audio tokenizer converts the audio into tokens, and the chat template marks the audio position with `<|audio|>` tokens. The encoder emits one audio token every 80 ms, which yields 12.5 tokens per second. So let’s say, you have an audio of 10 seconds, the number of input audio tokens would be 125. The encoder itself is a convolutional pyramid where each stage consists of a patch embedder of different patch sizes and 2 convolutional blocks of 1d kernels of size 7 with stride of 1. These stages are repeated for patch sizes of \\[16, 6, 5, 4\\]. At the end of the final stage, the temporal frames are fed to a final block of RMSNorm → linear projection → RMSNorm.\n\n### Long Context\n\nThe context window is 1M tokens at inference. The configuration sets `max_position_embeddings` to 1,048,576. This includes image, video, and audio tokens in the same window as text.\n\nGrouped-query attention keeps the KV cache small for a model of this size. Each token adds 2 × 16 heads × 64 dimensions × 62 layers = 126,976 values to the cache. In BF16, this is about 250 KB per token and about 266 GB for a full 1M-token context.\n\n## Inference Support\n\n### Transformers\n\nThe Muse Spark Open implementation provides `MuseSparkForConditionalGeneration` for text generation from multimodal inputs and `MuseSparkProcessor` for preparing conversations. The processor combines the tokenizer, image and video processors, audio feature extractor, and chat template.\n\nUse a `DistributedConfig` to configure the parallelism settings. `tp_size` is the number of devices across which the dense layers are sharded, and `ep_size` is the number of devices across which the experts are distributed.\n\nYou can initialize the model and processor as follows.\n\n```py\nimport os\n\nfrom transformers import AutoModelForMultimodalLM, AutoTokenizer, DistributedConfig\n\ndistributed_config = DistributedConfig(\n tp_size=int(os.environ[\"WORLD_SIZE\"]),\n ep_size=int(os.environ[\"WORLD_SIZE\"]),\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"meta-models/Muse-Spark-Open-816B-A42B\n\")\nmodel = AutoModelForMultimodalLM.from_pretrained(\n \"meta-models/Muse-Spark-Open-816B-A42B\",\n distributed_config=distributed_config,\n)\n\nmessages = [\n {\"role\": \"user\", \"content\": \"Explain in one sentence why the sky is usually blue.\"},\n]\ninputs = tokenizer.apply_chat_template(\n messages,\n reasoning_effort=\"medium\", # default: xhigh, options: low/medium/high/xhigh\n add_generation_prompt=True,\n tokenize=True,\n return_dict=True,\n return_tensors=\"pt\",\n).to(model.device).to(model.dtype)\n\ninput_len = inputs[\"input_ids\"].shape[-1]\n\noutputs = model.generate(**inputs, max_new_tokens=2000)\n```\n\n#### Text-only Inference\n\nPrepare the conversation, then pass the messages to `processor.apply_chat_template`, enabling `add_generation_prompt=True`, `tokenize=True`, `return_dict=True`, and `return_tensors=\"pt\"`. Pass the processor output to `model.generate` and decode the newly generated tokens, excluding the input prefix. The processor has a lever for the reasoning effort, mapping symbolic levels (low/medium/high/xhigh) to the integers the model itself accepts (16, 64, 256, 512). Note that turning off reasoning is not possible with Muse Spark Open, so keep the model reasoning at “low” if you want to consume less tokens to solve an easier task. You can also set a numeric effort by including a line like `Reasoning strength: 1` in the system prompt.\n\n```py\nmessages = [\n {\"role\": \"system\", \"content\": \"You should only answer with a number.\"},\n {\"role\": \"user\", \"content\": \"What is 17 * 23?\"},\n]\n\ninputs = processor.apply_chat_template(\n messages,\n add_generation_prompt=True,\n tokenize=True,\n return_dict=True,\n return_tensors=\"pt\",\n reasoning_effort=\"high\", # options:low/medium/high/xhigh\n).to(model.device).to(model.dtype)\n\noutput = model.generate(**inputs, max_new_tokens=2000)\ngenerated_tokens = output[0][inputs[\"input_ids\"].shape[1] :]\nprint(processor.decode(generated_tokens, skip_special_tokens=False))\n\n```\n\n#### Images and Text\n\nThe Muse Spark Open processor expands each image marker into the number of patch tokens required by the processed image, surrounded by image boundary tokens. This expansion keeps the text positions aligned with the visual features passed to the model. The processor also handles multiple images within a single prompt.\n\n```py\nimage_url = (\n \"https://huggingface.co/datasets/merve/vl-test-suite/\"\n \"resolve/main/pills.jpg\"\n)\nmessages = [\n {\n \"role\": \"user\",\n \"content\": [\n {\n \"type\": \"image\",\n \"image\": image_url,\n },\n {\n \"type\": \"text\",\n \"text\": \"Do any of the components in this supplement interact?\",\n },\n ],\n },\n]\n\ninputs = processor.apply_chat_template(\n messages,\n tokenize=True,\n add_generation_prompt=True,\n reasoning_effort=\"medium\",\n return_dict=True,\n return_tensors=\"pt\",\n).to(model.device).to(model.dtype)\ninput_len = inputs[\"input_ids\"].shape[-1]\n\noutputs = model.generate(**inputs, max_new_tokens=2000)\nresponse = processor.decode(outputs[0][input_len:], skip_special_tokens=False)\n\nprocessor.parse_response(response)\n```\n\n#### Video Inference\n\nVideo prompts include frame timing as well as visual content. The processor handles videos with audios in them (which is an upgrade from Muse Glimmer thanks to the audio encoder). The processor downsamples video frames and audio to align frames together. By default, videos are sampled to 2 FPS, capped at 96 frames. You can modify this by setting the fps of the video processor:\n\n```\nprocessor.video_processor.fps = 4.0\n```\n\nAs a general rule of thumb, videos are frames in temporal sequence, and the more you downsample, the more you will lose frames in between specific events, so you need to keep fps high if you want more accurate generations. However, there’s a trade-off because you will require more memory and increase latency for prefill. The frames are interleaved with text input that states timestamps (e.g. “Time: 3.5s”). To enable passing in videos with audio, pass `load_audio_from_video=True` to `apply_chat_template`.\n\nPreserve the video's metadata when preparing inputs with `return_metadata=True` so the processor can assign the correct framerate.\n\n```py\nimport torch\n\nmessages = [\n {\n \"role\": \"user\",\n \"content\": [\n {\n \"type\": \"text\",\n \"text\": (\n \"Describe what happens in this video. \"\n \"Use timestamps and include relevant speech or sounds.\"\n ),\n },\n {\n \"type\": \"video\",\n \"url\": \"https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/ramen.mp4\"\n },\n ],\n }\n]\n\ninputs = processor.apply_chat_template(\n messages,\n add_generation_prompt=True,\n tokenize=True,\n return_dict=True,\n return_tensors=\"pt\",\n load_audio_from_video=True,\n return_metadata=True,\n).to(model.device).to(model.dtype)\n\nwith torch.inference_mode():\n output_ids = model.generate(\n **inputs,\n max_new_tokens=256,\n do_sample=False,\n )\n\ngenerated_ids = output_ids[:, inputs[\"input_ids\"].shape[1] :]\nresponse = processor.batch_decode(\n generated_ids,\n skip_special_tokens=True,\n)[0]\n\nprint(response)\n```\n\n### Audio Inference\n\nThe Muse Spark Open audio tower consumes a raw waveform. The processor expects mono audio at 24 kHz and pads the waveform to a multiple of 1,920 samples, corresponding to one audio embedding per 80 milliseconds. Audio supplied directly to the processor at another sample rate must first be resampled. You can do this by using the `load_audio` util of transformers. `load_audio` uses torchcodec, librosa or torchaudio backends, so make sure to have one of these installed locally before running inference.\n\nIf you want to simply use `apply_chat_template` with messages though, this is all handled for you under the hood:\n\n```py\nimport torch\nfrom transformers import AutoProcessor, AutoModelForMultimodalLM\n\nmodel_id = \"meta-models/Muse-Spark-Open-816B-A42B\"\nprocessor = AutoProcessor.from_pretrained(model_id)\n\nmodel = AutoModelForMultimodalLM.from_pretrained(model_id).eval()\n\nmessages = [\n {\n \"role\": \"user\",\n \"content\": [\n {\n \"type\": \"audio\",\n \"url\": \"https://huggingface.co/buckets/huggingface/audio-samples/resolve/bbaudio_question_39.mp3\",\n },\n {\n \"type\": \"text\",\n \"text\": \"Answer the question.\",\n },\n ],\n }\n]\n\n# Downloads the audio, decodes it as mono, resamples it to 24 kHz, pads it, creates input_values_mask, and expands the audio placeholders.\ninputs = processor.apply_chat_template(\n messages,\n add_generation_prompt=True,\n tokenize=True,\n return_dict=True,\n return_tensors=\"pt\",\n load_audio_backend=\"auto\",\n).to(model.device).to(model.dtype)\n\nwith torch.inference_mode():\n output_ids = model.generate(\n **inputs,\n max_new_tokens=128,\n do_sample=False,\n )\n\n# Decode only newly generated tokens\nprompt_length = inputs[\"input_ids\"].shape[1]\ngenerated_text = processor.batch_decode(\n output_ids[:, prompt_length:],\n skip_special_tokens=True,\n)\n\nprint(generated_text[0])\n```\n\nThe processor inserts the matching number of audio tokens between audio boundary markers. It also returns a mask identifying real samples, allowing the model to exclude padded audio embeddings.\n\n#### Tool Calling\n\nMuse Spark Open can do text-only and multimodal tool calling. You can make the model call tools as follows.\n\n```py\n\nfrom transformers import AutoProcessor, AutoModelForMultimodalLM\n\nMODEL_ID = \"meta-models/Muse-Spark-Open-816B-A42B\"\n\n# Load model\nprocessor = AutoProcessor.from_pretrained(MODEL_ID)\nmodel = AutoModelForMultimodalLM.from_pretrained(\n MODEL_ID,\n dtype=\"auto\",\n device_map=\"auto\"\n)\n\ntools = [\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"weather.get\",\n \"description\": \"Get the current weather for a city.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"city\": {\"type\": \"string\"},\n },\n \"required\": [\"city\"],\n },\n },\n }\n]\n\n\nmessages = [\n {\n \"role\": \"user\",\n \"content\": [\n {\"type\": \"image\", \"image\": \"https://huggingface.co/datasets/merve/vl-test-suite/resolve/main/SF.png\"},\n {\"type\": \"text\", \"text\": \"I'm going to the city in this picture. What clothes should I wear?\"},\n ],\n },\n]\n\ninputs = processor.apply_chat_template(\n messages,\n tools=tools,\n tokenize=True,\n return_dict=True,\n return_tensors=\"pt\",\n add_generation_prompt=True,\n reasoning_strength=\"high\"\n).to(model.device).to(model.dtype)\n\ninput_len = inputs[\"input_ids\"].shape[-1]\noutputs = model.generate(**inputs)\nresponse = processor.parse_response(\n completion,\n prefix=inputs[\"input_ids\"][0],\n tools=tools,\n )\nprint(response)\n```\n\nMuse Spark Open uses a channel-based chat template (Harmony, first introduced by gpt-oss). Note that you don’t have to know about the internals, but it’s useful to check the channels the model routes to, in case you get low quality outputs. In the input message below:\n\n```\n<|begin_of_text|>\n <|start|>system<|message|>\n You are a helpful AI assistant.\n Knowledge cutoff: 2026-01-04.\n Current date: ...\n ...\n <|eot|>\n <|start|>user<|message|>...\n <|eot|>\n <|start|>assistant\n```\n\n`<|start|>assistant` is intentionally left open, it’s the model that chooses recipient with `to=` . The model has different recipients, e.g. `self` asks the model to reason,\n\n```\nto=self<|message|>the model reasons here<|eom|> # reasoning\n<|start|>assistant to=user<|message|>final answer<|eot|> # final answer\n```\n\n`self` is for reasoning, `commentary` are comments before and between tool calls visible to the end user, `user` is the final response.\\\nTool-call routing is parsed as follows (assuming you have passed `tools` to `apply_chat_template`:\n\n```\n<|start|>assistant to=weather.get<|message|>\n \n \n San Francisco\n \n \n <|eot|>\n```\n\n### Llama.cpp\n\nMuse Spark 2-bit running on Llama Server\n\nMuse Spark Open is integrated in llama.cpp and comes in the prebuilt binaries. You can install this with following command:\n\n```\ncurl -LsSf https://llama.app/install.sh | sh\n```\n\nYou can start the server as follows with quant suffix attached to model ID:\n\n```\n\nllama serve -hf ggml-org/Muse-Spark-Open-816B-A42B-GGUF:Q2_K\n```\n\nThe repository containing quants is [here](https://huggingface.co/ggml-orgf/Muse-Spark-Open-816B-A42B-GGUF). Once the server has started, you can head to localhost:9931 to chat with the built-in WebUI. From the web UI you can add audio and images for inference and adjust the reasoning lever.\\\nNote that turning off reasoning is not possible with Muse Spark Open, so keep the model reasoning at “low” if you want to consume less tokens to solve an easier task.\\\nWe tested quants on Mac M3 Ultra (512GB). An imatrix-calibrated 2-bit quant runs at 18 tokens/s, with plenty of room free for the KV cache (304 GB used by the model).\n\nYou can also query the server as follows. Feel free to check out [the API reference](https://llama.app/docs/serve) for multimodality, tool calls and more.\n\n```\n## Text-only inference\ncurl http://localhost:8080/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"messages\": [\n {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n {\"role\": \"user\", \"content\": \"Write a limerick about python exceptions\"}\n ]\n }'\n\n## Image-text inference\ncurl -s http://127.0.0.1:8080/v1/chat/completions \\\n -H \"Content-Type: application/json\" -H \"Authorization: Bearer \" \\\n -d '{\"model\":\"muse-spark-open\",\"messages\":[{\"role\":\"user\",\"content\":[\n {\"type\":\"text\",\"text\":\"What is in this image?\"},\n {\"type\":\"image_url\",\"image_url\":{\"url\":\"data:image/png;base64,...\"}}\n ]}],\"temperature\":0}'\n\n## Audio-text inference\ncurl -s http://127.0.0.1:8080/v1/chat/completions \\\n -H \"Content-Type: application/json\" -H \"Authorization: Bearer \" \\\n -d '{\"model\":\"muse-spark-open\",\"messages\":[{\"role\":\"user\",\"content\":[\n {\"type\":\"text\",\"text\":\"Transcribe this audio.\"},\n {\"type\":\"input_audio\",\"input_audio\":{\"data\":\"data:audio/wav;base64,...\"}}\n ]}],\"temperature\":0}'\n```\n\nllama.cpp team ships different quantized versions. We have tested Q2\\_K, Q4\\_K\\_S and Q\\_8 on Apple Silicon chips. You can run each quant by adding the quant id as suffix, `llama serve -hf ggml-org/Muse-Spark-Open-816B-A42B:Q2_K`.\n\n### SGLang\n\nThe Muse Spark Open integration for SGLang supports text, image, audio, and video inputs. The NVFP4 reference configuration runs across four GB300 GPUs.\n\nFrom the SGLang integration environment, download the checkpoint from the Hugging Face Hub and start the server:\n\n```\nexport MODEL_ID=meta-models/Muse-Spark-Open-816B-A42B-NVFP4\nexport CKPT=\"$(hf download \"$MODEL_ID\")\"\n\nSGLANG_FLASHINFER_NVFP4_PER_TOKEN_ACTIVATION=1 \\\nuv run sglang_meta serve \\\n --model-path \"$CKPT\" \\\n --served-model-name \"$MODEL_ID\" \\\n --quantization modelopt_fp4 \\\n --moe-runner-backend flashinfer_trtllm \\\n --tp-size 4 \\\n --ep-size 4 \\\n --port 30000\n```\n\nThe checkpoint includes the chat template, which we pass explicitly. The launcher configures the Muse reasoning and tool-call parsers.\n\nSend a request through the OpenAI-compatible API:\n\n```\ncurl http://localhost:30000/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"meta-models/Muse-Spark-Open-816B-A42B-NVFP4\",\n \"messages\": [{\n \"role\": \"user\",\n \"content\": \"Explain how a mixture-of-experts model works.\"\n }],\n \"max_tokens\": 2048\n }'\n```\n\nThe response separates the reasoning trace in `reasoning_content` from the final answer in `content`.\n\n### vLLM\n\nThe Muse Spark Open reference integration for vLLM supports text and image inputs with the NVFP4 checkpoint that require blackwell series GPUs.\n\nAfter installing the reference plugin and its dependencies, download the checkpoint from the Hugging Face Hub and launch the server:\n\n```\nexport MODEL_ID=meta-models/Muse-Spark-Open-816B-A42B-NVFP4\nexport CKPT=\"$(hf download \"$MODEL_ID\")\"\n\nVLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \\\nVLLM_PLUGINS=musespark1x_models,ptq_nvfp4 \\\nvllm serve \"$CKPT\" \\\n --served-model-name \"$MODEL_ID\" \\\n --tokenizer-mode musespark1x \\\n --config-format auto \\\n --load-format internal_safetensor \\\n --tensor-parallel-size 4 \\\n --enable-expert-parallel \\\n --max-model-len 131072 \\\n --gpu-memory-utilization 0.92 \\\n --reasoning-parser musespark1x \\\n --tool-call-parser musespark1x \\\n --enable-auto-tool-choice \\\n --port 8000\n```\n\nThis configuration sets a 128K-token serving context. The plugin supplies the model implementation, checkpoint loader, and reasoning and tool-call parsers.\n\nSend a request using the same Hub model ID:\n\n```\ncurl http://localhost:8000/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"meta-models/Muse-Spark-Open-816B-A42B-NVFP4\",\n \"messages\": [{\n \"role\": \"user\",\n \"content\": \"Explain how a mixture-of-experts model works.\"\n }],\n \"max_tokens\": 2048\n }'\n```\n\n## MLX\n\nMuse Spark Open is available in [oMLX](https://github.com/jundot/omlx) and [mlx-lm](https://github.com/ml-explore/mlx-lm) today. oQ quants optimized for oMLX are available in [3-bit](https://huggingface.co/mlx-community/Muse-Spark-Open-816B-A42B-oQ3e) and [4-bit](https://huggingface.co/mlx-community/Muse-Spark-Open-816B-A42B-oQ4e), ready to use right away.\n\nWe tested the oQ3e quant on an M3 Ultra (512 GB). The model takes about 344 GB, leaving room for the KV cache, and runs at about 225 tok/s prefill and 21 tok/s generation with a 1024-token prompt and 128 generated tokens.\n\nOnce you download the model through the oMLX interface, you can chat with it in the built-in chat UI. From the chat UI you can add images and adjust the reasoning effort on the fly, from low to xhigh. The model’s reasoning shows up as a collapsible thinking block, separate from the final answer.\n\nNote that it's not possible to turn reasoning off, and the default (Auto) uses xhigh. Set it to “low” if you want to use fewer tokens on easier tasks. [](https://huggingface.co/datasets/merve/meta-muse-spark-open-assets/resolve/main/omlx_muse_spark_open_reasoning_screenshot.png)\n\nThe language model is also available in [this PR](https://github.com/ml-explore/mlx-lm/pull/1975) of the official mlx-lm repository. You can run it on the same high-quality oQ3e quant like this:\n\n```\nmlx_lm.generate --model mlx-community/Muse-Spark-Open-816B-A42B-oQ3e \\\n --prompt \"Explain convolutional networks like I am 5\" \\\n --max-tokens 5000\n```\n\nThis is a big model, even the 3-bit quantization requires more than 256 GB of RAM, as explained above. You can also run distributed inference across multiple computers, please refer to the documentation for details.\n\n## Deploying Muse Spark Open with Inference Endpoints\n\nYou can deploy Muse Spark Open on Inference Endpoints with a pre-validated SGLang recipe for the NVFP4 variant which runs on either 8 × NVIDIA RTX Pro 6000 98 GB with 131072 context length, or with 8 x NVIDIA H200 141GB if you want 1048576 context length instead.\n\nTo deploy, run one command with the Hugging Face CLI:\n\n```sh\nhf endpoints catalog deploy --repo meta-models/Muse-Spark-Open-816B-A42B-NVFP4 --name muse-spark\n```\n\nOr open [endpoints.huggingface.co/new/meta-models/Muse-Spark-Open-816B-A42B-NVFP4](https://endpoints.huggingface.co/new/meta-models/Muse-Spark-Open-816B-A42B-NVFP4) and select Create Endpoint.\n\nThe endpoint exposes an OpenAI-compatible API with both Chat Completions API at `/v1/chat/completions` and Responses API at `/v1/responses`.\n\n```sh\ncurl \"https://.endpoints.huggingface.cloud/v1/chat/completions\" \\\n -H \"Authorization: Bearer $HF_TOKEN\" \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"meta-models/Muse-Spark-Open-816B-A42B-NVFP4\",\n \"messages\": [{\n \"role\": \"user\",\n \"content\": [\n {\"type\": \"image_url\", \"image_url\": {\"url\": \"https://huggingface.co/datasets/merve/vl-test-suite/resolve/main/pills.jpg\"}},\n {\"type\": \"text\", \"text\": \"Do any of the components in this supplement interact?\"}\n ]\n }],\n \"chat_template_kwargs\": {\"enable_thinking\": false}\n }'\n```\n\nAny OpenAI-compatible client works with the same URL and your Hugging Face token as the API key.\n\n## Fine-tuning with TRL\n\nYou can fine-tune the 816B checkpoint with the TRL `SFTTrainer`. This section shows how we fine-tuned it with TRL on 8 nodes of 8 H100 GPUs. The recipe trains LoRA adapters on the attention layers. The experts stay frozen. The full script is in the [TRL examples](https://github.com/huggingface/trl/tree/main/examples/sft_muse_spark_open).\n\nFirst, we need to split the model across 64 GPU devices. Given the large size of the model, we need to parallelize it across multiple axes. We shard the model with FSDP and split the experts across GPUs, using `DistributedConfig`, which tells `transformers` how to split the model when it loads:\n\n```py\nfrom transformers.distributed import DistributedConfig\n\ndistributed_config = DistributedConfig(fsdp_size=64, ep_size=8)\n```\n\nIn the snippet above, `fsdp_size=64` shards the model across all 64 GPUs with FSDP2, and `ep_size=8` splits the 256 experts of each layer into 8 groups. Transformers sends each token to the GPU that holds its expert, so each GPU trains on its own slice of the batch.\n\nNow, we can configure the `SFTConfig` for our training run. We’ve performed a simple task here to adapt the model’s chat template on an open source dataset, but you could apply it to your own use case.\n\n```py\nimport torch\nfrom datasets import load_dataset\nfrom peft import LoraConfig\nfrom transformers.distributed import DistributedConfig\nfrom trl import SFTConfig, SFTTrainer\n\ntrain_dataset = load_dataset(\"allenai/tulu-3-sft-mixture\", split=\"train[:500000]\")\n\ntraining_args = SFTConfig(\n output_dir=\"muse-spark-lora-sft\",\n model_init_kwargs={\"dtype\": torch.bfloat16, \"distributed_config\": DistributedConfig(fsdp_size=64, ep_size=8)},\n per_device_train_batch_size=1,\n max_steps=500,\n max_length=2048,\n packing=True,\n save_only_model=True,\n logging_steps=1,\n report_to=\"trackio\",\n)\n\ntrainer = SFTTrainer(\n model=\"meta-models/Muse-Spark-Open-816B-A42B\",\n args=training_args,\n train_dataset=train_dataset,\n peft_config=LoraConfig(r=16, lora_alpha=32, target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\"]),\n)\ntrainer.train()\n```\n\nThe core parts here are `packing=True` which joins examples into 2048-token sequences without padding. A block-diagonal mask keeps the examples separate. `save_only_model=True` saves the adapter and no optimizer state, because the trainer cannot resume a model that was sharded at load time. `LoraConfig` targets the four attention projections and the 256 experts of each layer stay frozen.\n\nThen we can run this command across each node:\n\n```sh\ntorchrun --nnodes 8 --nproc_per_node 8 --rdzv_backend c10d --rdzv_endpoint \"$MASTER_IP:29700\" sft_muse_spark_open.py\n```\n\nWe trained for 500 steps, which took about 2 hours on 64 H100s, at about 8,000 tokens per second. Mean token accuracy rose from 72% to 84% and the loss fell from 1.26 to 0.57, which signals a good fine-tuning run.\n\n[](https://huggingface.co/datasets/merve/meta-muse-spark-open-assets/resolve/main/training_loss.png)\n\n[](https://huggingface.co/datasets/merve/meta-muse-spark-open-assets/resolve/main/token_acc.png)\n\n## Demos\n\nMuse Spark Open is a coding and agentic model. We wanted to see that for ourselves, so we tried a mixture of multi-modal and agentic tasks that use the model's image and audio inputs. Everything below is something you can run or fork.\n\n### 3D Asset Generator\n\nWe combined 2-bit quantized Muse Spark open (served on llama.cpp) with the Pi coding agent to generate and iteratively improve 3D assets in [Three.js](http://Three.js). You can find the prompt [here](https://huggingface.co/datasets/merve/meta-muse-spark-open-assets/resolve/main/pi-3d-prompt.txt), as the model is quantized it requires a bit more steering. We get 18 TPS on llama.cpp on Mac Ultra.\n\n[](https://huggingface.co/datasets/merve/meta-muse-spark-open-assets/resolve/main/pi-demo.png)\n\nWe initially describe the asset, and Pi uses the model to generate Three.js code for the asset, renders the result in a browser, and captures screenshots from multiple angles. It then sends those images back to the same model on llama server for visual feedback. Pi uses that feedback to edit the code and repeats the process.\n\n### Browser Interactions with OpenEnv and Hugging Face Sandbox\n\nWe tested Spark on computer use tasks and were amazed by the performance, so we have built a lean demo that also serves to teach you how a computer use agent works. You can also further customize this for your needs. This is both an educational example and something you can run in a few lines. You can find it [here](https://huggingface.co/docs/openenv/main/en/environments/helium_browser).\n\nMuse Spark NVFP4 Browser Agent with OpenEnv and HF Sandboxes\n\n[Hugging Face Sandboxes](https://huggingface.co/docs/huggingface_hub/en/guides/sandbox) are sandboxes built on Hugging Face Jobs for easy code execution. The demo consists of three components:\n\n1. Muse Spark Open served with Inference Endpoints,\n2. OpenEnv wrapping web driver actions with helium and selenium (click, type, scroll etc) defining browser environment’s reset, step, observation, termination interface,\n3. Chromium web browser\n\nwhere the remote sandbox contains the environment and the browser as illustrated below.\n\n[](https://huggingface.co/datasets/merve/meta-muse-spark-open-assets/resolve/main/browser_server.png)\n\nYou can easily try this with a few steps, described in the example. Start Inference Endpoint with Muse Spark Open (NVFP4 works well too!), add your endpoint URL as environmental variable, publish your OpenEnv server in Spaces and also provide that, then you can run the agent loop. Muse Spark Open sees the previous actions, system prompt, url and the screenshot of the browser to take an action, and nothing else (no DOM, no HTML code, nothing) so it’s a truly multimodal browser demo.\n\n### Call your Browser\n\nFinally, we wanted to push Spark’s agentic multimodality to the limits, so we set up [this space](https://huggingface.co/spaces/burtenshaw/muse-spark-call-your-browser) that combines the model’s audio, image, and agentic properties into a browser agent. We give the model a general description of a thing we need and it interacts with the integrated browser to find options on Amazon.\n\nMuse Spark Open takes the voice input and reasons over the type of product that meets the descriptions, searches for it on Amazon, and selects the most appropriate based on criteria.\n\n## Wrapping Up\n\nMuse Spark Open is a full-size open-weight model from the Muse series. Start with the [BF16](https://huggingface.co/meta-models/Muse-Spark-Open-816B-A42B) or [NVFP4](https://huggingface.co/meta-models/Muse-Spark-Open-816B-A42B-NVFP4) checkpoint, or try one of the great `llama.cpp quants`. If you build something with Muse Spark Open, share it, and tell us in the [repository’s discussions.](https://huggingface.co/meta-models/Muse-Spark-Open-816B-A42B/discussions)\n",
"dateCreated": "2026-10-15T18:36:21.248140220+00:00",
"date_created": "2026-10-15T00:00:00+00:00",
"date_modified": "2026-10-15T00:00:00+00:00",
"date_published": "2026-10-15T00:00:00+00:00",
"description": "Muse Spark Open is an open-weight model from Meta. The mixture of experts model brings text, images, audio, and video into a model with approximately 816 billion parameters, around 42 billion active per token, and a context window of up to…",
"external_url": "https://huggingface.co/blog/meta-muse-spark-open",
"git": {
"commit": "ebb0941c5c06a2ab2377342d607535262a3e969e",
"path": "items/huggingface-co/2026/10/2026-10-15-welcome-muse-spark-open-open-weight-frontier-multimodal.md",
"permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/huggingface-co/2026/10/2026-10-15-welcome-muse-spark-open-open-weight-frontier-multimodal.md"
},
"id": "urn:aggr:item:a2905235365c2757da452e13f96137f589ac3b31",
"isBasedOn": {
"@id": "https://huggingface.co/blog/meta-muse-spark-open"
},
"mainEntity": {
"@id": "https://huggingface.co/blog/meta-muse-spark-open",
"@type": "CreativeWork",
"archivedAt": {
"@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-10-15-welcome-muse-spark-open-open-weight-frontier-multimodal/#webpage"
},
"dateModified": "2026-10-15T00:00:00+00:00",
"datePublished": "2026-10-15T00:00:00+00:00",
"headline": "Welcome Muse Spark Open: open-weight frontier multimodal mixture-of-experts model from Meta",
"timeRequired": "PT19M",
"url": "https://huggingface.co/blog/meta-muse-spark-open",
"wordCount": 4253
},
"name": "Archived snapshot: Welcome Muse Spark Open: open-weight frontier multimodal mixture-of-experts model from Meta",
"preview": null,
"reading_minutes": 19,
"replicated_at": null,
"source": "huggingface-co",
"source_name": "Hugging Face - Blog",
"summary": null,
"tags": [],
"temporalCoverage": "2026-10-15T18:36:21.248140220+00:00",
"title": "Welcome Muse Spark Open: open-weight frontier multimodal mixture-of-experts model from Meta",
"url": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-10-15-welcome-muse-spark-open-open-weight-frontier-multimodal/",
"word_count": 4253
}