{ "@context": "https://schema.org", "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-10-29-introducing-anyspeechrecognizer-and-anydecisionmodel-open/#webpage", "@type": [ "WebPage", "ArchiveComponent" ], "articleBody": "Language models can do a lot. They write, summarize, translate, and call tools. But they're only a small part of what machine learning can do.\n\nSay a customer calls your support line. First, you want to know what they said (speech recognition). Then, you want to decide where to send them (decision). You *could* hand both to a language model, but there are models built for each that are better, faster, and less expensive.\n\nIn our [AnyLanguageModel 1.0 recap](https://huggingface.co/blog/anylanguagemodel-1-0), we looked at a year of local and hosted models in Swift. With AnyLanguageModel and swift-huggingface now on their 1.x releases, we're extending the same approach to speech and decisions: familiar Swift APIs with a choice of model.\n\nToday we're announcing two packages:\n\n- [AnySpeechRecognizer](https://github.com/mattt/AnySpeechRecognizer) implements Apple's new speech-to-text API with a choice of recognition engine, including NVIDIA's Parakeet and Alibaba's Qwen3-ASR running locally on Core ML, and adds speaker diarization to track who speaks when.\n- [AnyDecisionModel](https://github.com/mattt/AnyDecisionModel) answers yes-or-no, multiple-choice, and rating-scale questions with typed results and probabilities. It derives each answer from the model's probabilities instead of generating text, using a model on your device with MLX, Core AI, or llama.cpp, or a decision model from Jev or Ollama.\n\nBoth packages can download open models from the Hugging Face Hub and run them on device.\n\n## AnySpeechRecognizer\n\nComputers have been transcribing speech for decades, and in the past few years, they've gotten very good at it. Apple platforms already have an API for it. macOS 26 and iOS 26 introduced `SpeechAnalyzer`, Apple's new speech-to-text API. You add modules such as `SpeechTranscriber` and `SpeechDetector` to an analyzer, give it a file or a stream of audio buffers, and read results from async sequences as they arrive. `AssetInventory` installs the models, so your app decides when downloads happen.\n\nIt's a well-designed API, but the only engine behind it is Apple's. AnySpeechRecognizer keeps the API and lets you choose the engine:\n\n```diff\n- import Speech\n+ import AnySpeechRecognizer\n```\n\nThe package mirrors `SpeechAnalyzer`, `SpeechTranscriber`, `DictationTranscriber`, `SpeechDetector`, and `AssetInventory`, with Apple's Speech framework as the default engine. If your code uses that shared API, changing the import is enough. To use Parakeet instead, change only how you create the transcriber:\n\n```swift\nlet transcriber = SpeechTranscriber(\n model: ParakeetSpeechModel(variant: .v3),\n transcriptionOptions: [],\n reportingOptions: [.volatileResults],\n attributeOptions: [.audioTimeRange]\n)\n```\n\nThe steps for installing assets, running the analyzer, and reading results stay the same. The [README](https://github.com/mattt/AnySpeechRecognizer#transcribe-a-file) has a complete example and [lists the differences](https://github.com/mattt/AnySpeechRecognizer#differences-from-apples-speech-framework) from Apple's framework.\n\n### Meet the models\n\n**Parakeet v3** is NVIDIA's [Parakeet TDT 0.6B v3](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3). It's a polyglot: it recognizes 25 European languages and works out which one it's hearing, so you don't pass a locale.\n\n**Parakeet Ultra** is Moondream's [post-trained take on v3](https://huggingface.co/moondream/parakeet-ultra). It was more accurate in most of our measurements.\n\n**Parakeet Unified** is NVIDIA's [English-only Parakeet](https://huggingface.co/nvidia/parakeet-unified-en-0.6b).\n\nAll three Parakeets run on the package's own Core ML runtime, which supports file and live input, volatile results, word timing, and word confidence.\n\n**Qwen3-ASR** is Alibaba's speech model, in [0.6B](https://huggingface.co/Qwen/Qwen3-ASR-0.6B-hf) and [1.7B](https://huggingface.co/Qwen/Qwen3-ASR-1.7B-hf) sizes. It recognizes 30 languages and 22 Chinese dialects, including Mandarin, Cantonese, Japanese, and Korean, which Parakeet doesn't cover. It detects the language automatically, or you can set one:\n\n```swift\nlet transcriber = SpeechTranscriber(\n model: QwenSpeechModel(variant: .large, language: Locale.Language(identifier: \"yue\")),\n transcriptionOptions: [],\n reportingOptions: [],\n attributeOptions: []\n)\n```\n\nQwen3-ASR gives one final result for each segment of up to 40 seconds, without volatile results, word timing, or confidence. It runs on the GPU and is slower than Parakeet, so it's better suited to transcribing files than live input.\n\nTo add another engine, implement `SpeechRecognitionModel` for its configuration and `SpeechRecognitionExecutor` for one module's session. The README includes a [small provider](https://github.com/mattt/AnySpeechRecognizer#custom-providers) that reports placeholder text to help you get started.\n\n### Core ML models on the Hub\n\nTo run these models in apps, we converted them to Core ML and published them on the Hub:\n\n| Model | Core ML conversion | Original checkpoint | License |\n| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |\n| Parakeet TDT 0.6B v3 | [mattt/parakeet-tdt-0.6b-v3-coreml](https://huggingface.co/mattt/parakeet-tdt-0.6b-v3-coreml) | [nvidia/parakeet-tdt-0.6b-v3](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3) | CC BY 4.0 |\n| Parakeet Ultra | [mattt/parakeet-ultra-coreml](https://huggingface.co/mattt/parakeet-ultra-coreml) | [moondream/parakeet-ultra](https://huggingface.co/moondream/parakeet-ultra) | CC BY 4.0 |\n| Parakeet Unified EN 0.6B | [mattt/parakeet-unified-en-0.6b-coreml](https://huggingface.co/mattt/parakeet-unified-en-0.6b-coreml) | [nvidia/parakeet-unified-en-0.6b](https://huggingface.co/nvidia/parakeet-unified-en-0.6b) | [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/) |\n| Qwen3-ASR 0.6B | [mattt/qwen3-asr-0.6b-coreml](https://huggingface.co/mattt/qwen3-asr-0.6b-coreml) | [Qwen/Qwen3-ASR-0.6B-hf](https://huggingface.co/Qwen/Qwen3-ASR-0.6B-hf) | Apache 2.0 |\n| Qwen3-ASR 1.7B | [mattt/qwen3-asr-1.7b-coreml](https://huggingface.co/mattt/qwen3-asr-1.7b-coreml) | [Qwen/Qwen3-ASR-1.7B-hf](https://huggingface.co/Qwen/Qwen3-ASR-1.7B-hf) | Apache 2.0 |\n| Streaming Sortformer 4spk v2 | [mattt/diar-streaming-sortformer-4spk-v2-coreml](https://huggingface.co/mattt/diar-streaming-sortformer-4spk-v2-coreml) | [nvidia/diar\\_streaming\\_sortformer\\_4spk-v2](https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2) | CC BY 4.0 |\n| Streaming Sortformer 4spk v2.1 | [mattt/diar-streaming-sortformer-4spk-v2.1-coreml](https://huggingface.co/mattt/diar-streaming-sortformer-4spk-v2.1-coreml) | [nvidia/diar\\_streaming\\_sortformer\\_4spk-v2.1](https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2.1) | [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/) |\n\nEach repository records its source checkpoint and a checksum for every file, and the package pins it to a fixed revision. The Parakeet repositories add an 8-second encoder for live input, which cuts live analysis time by about a third at the cost of a somewhat higher word error rate. The repository also has the conversion workflows, most of which follow FluidInference's [mobius](https://github.com/FluidInference/mobius) scripts, and checks each conversion against the original model.\n\nDownloads happen only when your app asks for them: about 600 MB for each Parakeet, plus about 590 MB for the live encoder, 0.9 to 2.4 GB for Qwen3-ASR, and about 250 MB for Sortformer. Files go in the standard Hugging Face cache, so a Mac app outside the sandbox shares them with Python and other Hugging Face tools.\n\n### Speaker diarization\n\nWhen you transcribe a conversation, you often want to know who said what. That's speaker diarization, and Apple's Speech framework doesn't have a module for it, so AnySpeechRecognizer adds one. `SpeakerDiarizer` runs NVIDIA's [Streaming Sortformer](https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2) on Core ML alongside an Apple or Parakeet transcriber, on the same timeline:\n\n```swift\nlet diarizer = SpeakerDiarizer(model: SortformerDiarizationModel())\nif let request = try await AssetInventory.assetInstallationRequest(\n supporting: [transcriber, diarizer]\n) {\n try await request.downloadAndInstall()\n}\nlet analyzer = SpeechAnalyzer(modules: [transcriber, diarizer])\n```\n\nThe diarizer tracks up to four speakers, even when they talk over each other. A `SpeakerTimeline` collects its results and attributes each timed word to a speaker, so you can print a transcript by speaker turn. Like image support in AnyLanguageModel, this goes beyond what Apple's API offers today.\n\nThe diarizer uses Sortformer v2 by default. You can also choose [v2.1](https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2.1), which NVIDIA trained with more meeting data and which was more accurate on meetings in our measurements, too.\n\n### How well does it work?\n\nWe measured the package with its `hear` command-line tool on an M4 Max Mac Studio with macOS 27.0.1. Here are the word error rates on LibriSpeech files of 3 to 8 minutes:\n\n| Transcriber | test-clean | test-other |\n| ------------------------- | ---------- | ---------- |\n| Parakeet v3, file | 1.55% | 2.78% |\n| Parakeet v3, live | 4.73% | 4.29% |\n| Parakeet Ultra, file | 1.66% | 2.55% |\n| Parakeet Ultra, live | 2.65% | 3.28% |\n| Parakeet Unified, file | 1.43% | 2.19% |\n| Parakeet Unified, live | 1.45% | 2.42% |\n| Apple's transcriber, file | 2.12% | 3.97% |\n\nParakeet transcribes an hour of audio in about 20 to 40 seconds from a file, or about 1.5 to 2.5 minutes live, with about 100 MB of memory once Core ML has compiled the model. Qwen3-ASR is much slower and needs a few gigabytes of memory. Meetings are hard for everyone: on AMI recordings, where people talk over each other, every engine we tested had a word error rate above 23%.\n\nThese numbers come from one machine and a few datasets, so measure with your own hardware and audio. `hear evaluate` works on your own files, and the repository's `natural-speech-eval.py` tool [reproduces these measurements](https://github.com/mattt/AnySpeechRecognizer#accuracy-and-performance). So far, we've tested only on macOS 27, not on an iPhone.\n\nNext, we'd like to add more engines, including Core AI on macOS 27 and iOS 27.\n\n## AnyDecisionModel\n\nAnyDecisionModel is a different story. There's no Apple framework to map onto, and decision models aren't new, exactly.\n\n> It's only a decision model if it comes from the Décision region of France; otherwise, it's sparkling zero-shot classification.\n\nWhat's new is the name, a lot of hype, and a crop of models trained to make decisions. We think that's where things are going, so we wanted a Swift API that's ready for them.\n\nApps often ask language models to make decisions. *Is this message urgent?* *Which team should handle this ticket?* *How frustrated is this customer?* The usual approach is to prompt a language model and parse the text that it generates. That gives you one answer, but doesn't show the probabilities it assigned to the alternatives.\n\nHere's what a decision looks like in Swift:\n\n```swift\nimport AnyDecisionModel\n\nenum Department: String, Choosable {\n case billing, technical, sales\n}\n\nlet ticket = \"I was charged twice for my order, and now the app logs me out when I try to ask for a refund.\"\n\nlet model = MLXDecisionModel(modelID: \"mlx-community/Qwen3-4B-Instruct-2507-4bit\")\nlet session = DecisionSession(model: model, state: .text(ticket))\n\nlet urgent = try await session.probability(of: \"Does this convey urgency?\")\nlet team = try await session.choice(\"Which team?\", from: Department.self)\nlet anger = try await session.score(\n \"How frustrated?\",\n levels: [\"Calm\", \"Frustrated\", \"Very angry\"]\n)\n```\n\n`urgent` is a `Double`. `team` is a `Choice` containing the selected case, a probability for each option, and a confidence value. `anger` is a `Score` containing the expected level, a probability for each level, and a confidence value. Because `Department` conforms to `Choosable`, `team.value` is a `Department` that you can use in an exhaustive `switch`.\n\nIf you've used Foundation Models or AnyLanguageModel, the model-and-session pattern will look familiar. A `DecisionSession` holds a single state, which can be text or JSON, and every question is about that state. Questions are independent: an answer never becomes input to a later question. To ask several questions at once, pass them to `decide(_:)` as a batch.\n\nFor each question, the MLX backend builds a prompt with a one-token label for each allowed answer, then reads the probability of each label from one forward pass.\n\n### Try it out: Emoji Magnet\n\nEmoji Magnet is a macOS app with a text field above a pile of 343 emoji. Type a phrase, such as \"things you wear\", and the emoji that match rise out of the pile. Refine it to \"things you wear in winter\", and the emoji that no longer match fall back.\n\nThe app has no built-in keyword rules; the model decides everything at inference time. When you pause typing, it asks the model one yes-or-no question for each emoji, for example:\n\n```\nStatement: a heavy winter coat fits the description \"things you wear\". Is this statement true?\n```\n\nThe session's state holds the phrase. The demo above uses the hosted Jev backend, which answers all 343 questions in one request, so the emoji settle in about half a second for each phrase. Without a `TYPESAFE_AI_API_KEY`, the app runs Qwen3 4B on your Mac with the MLX backend. That backend uses prefix caching to evaluate the state once, then answers 16 questions per forward pass. On an M4 Max, a group of 16 takes about 0.45 seconds after the model loads. A full pass over all 343 emoji takes about 10 seconds, and each group's results appear as soon as they arrive.\n\nEmoji Magnet is in the [AnyDecisionModel-Examples](https://github.com/mattt/AnyDecisionModel-Examples) repository.\n\n### Decision heads and Clef\n\nReading next-token probabilities works with general-purpose language models, but those models weren't trained to make decisions. Cloudflare's [Clef](https://huggingface.co/Cloudflare/clef) models are an example of a newer approach: a model with a trained decision head that turns a state and a schema of typed questions into decisions, returning probabilities for all allowed options across the questions in one forward pass. [Clef-Flash](https://huggingface.co/Cloudflare/clef-flash) is the 9B variant.\n\nAnyDecisionModel runs Clef-Flash on a Mac with MLX:\n\n```swift\nlet model = MLXHeadDecisionModel(\n modelID: \"mlx-community/clef-flash-4bit\",\n head: .jointSchema\n)\n```\n\nYou use the same sessions and questions, but `MLXHeadDecisionModel` reads each decision from Clef's trained decision head instead of from next-token probabilities. The [4-bit model](https://huggingface.co/mlx-community/clef-flash-4bit) is a 6.2 GB download, and its model card recommends a Mac with at least 16 GB of memory. Clef can read images and video, but this backend accepts only text and JSON states.\n\n### Calibrating and checking probabilities\n\nRaw probabilities from language models are often overconfident. The local backends accept a `Calibration`, which applies a temperature to every answer and a bias to yes-or-no answers. Use separate datasets to fit the calibration and evaluate the results.\n\nLanguage models can also favor an option because of its position in a list. On the MLX, Core AI, and llama.cpp next-token backends, turn on `rotationDebiasing` to ask a choice question once for each rotation of the options and average the probabilities. That costs one prompt evaluation per rotation.\n\nEach local choice and score also includes a confidence value based on the entropy of its probabilities. It measures how concentrated the probabilities are, not whether the answer is right, so check the error rate on your own data before you pick a threshold. The next-token backends' diagnostics report how much probability the model assigned to the allowed answers. A low value means it assigned most of the probability elsewhere.\n\n### Backends\n\nAnyDecisionModel has six backends:\n\n- **MLX** runs a small language model on Apple silicon with [mlx-swift-lm](https://github.com/ml-explore/mlx-swift-lm) and downloads it from the Hub.\n- **MLX decision heads** run a model with its own trained decision head, such as Clef.\n- **Core AI** does the same as MLX with a Core AI model on macOS 27 and iOS 27.\n- **llama.cpp** reads decisions from a local GGUF model through [llama.swift](https://github.com/mattt/llama.swift).\n- **Jev** calls TypeSafe's Jev through the [System One API](https://docs.typesafe.ai/api/system-one), or any server that implements it, such as [Kev](https://github.com/jaredpalmer/kev).\n- **Ollama** calls a System One decision model on a local Ollama server or in Ollama's cloud. It requires Ollama 0.35.0 or later, or 0.40.5 or later for cloud models.\n\nThe next-token backends label answers \"yes\" and \"no\", A to Z, or 0 to 9, so a choice can have at most 26 options, and a score at most 10 levels. Decision heads have no such limit.\n\nLike AnyLanguageModel, the package uses Swift package traits. Without traits, it has no third-party dependencies. You get the core types and the Jev and Ollama backends, which also build on Linux. Enable the `MLX`, `CoreAI`, or `Llama` trait for the local backends.\n\n## Get started\n\nBoth packages require Swift 6.2 or later and macOS 26 or iOS 26 or later. The Parakeet and Sortformer runtimes and the MLX backend require Apple silicon. AnyDecisionModel's MLX build tasks use Xcode's Swift 6.4 or later and the Metal toolchain; Core AI requires Xcode 27 and an OS 27 runtime.\n\nAdd the packages to your `Package.swift`:\n\n```swift\ndependencies: [\n .package(url: \"https://github.com/mattt/AnySpeechRecognizer\", from: \"0.1.0\"),\n .package(\n url: \"https://github.com/mattt/AnyDecisionModel\",\n from: \"0.6.0\",\n traits: [\"MLX\"]\n ),\n]\n```\n\nFor now, AnyDecisionModel's `CoreAI` trait can't be combined with AnySpeechRecognizer, because coreai-kit 0.7.3 requires swift-huggingface below 1.0 and AnySpeechRecognizer requires 1.x.\n\nEach README has more examples and details about requirements and backends. AnySpeechRecognizer also includes a command-line tool, `hear`, that transcribes local files, labels their speakers, and evaluates the results. Every `hear` command requires `--backend apple`, `--backend parakeet`, or `--backend qwen`; run `hear prepare` with the same backend and model options before analysis to install the needed assets.\n\nTry them in your apps and tell us what works and what doesn't. Issues and pull requests are welcome on both repositories.\n\n## Links\n\n- [AnySpeechRecognizer on GitHub](https://github.com/mattt/AnySpeechRecognizer)\n- [AnyDecisionModel on GitHub](https://github.com/mattt/AnyDecisionModel)\n- [AnyDecisionModel-Examples on GitHub](https://github.com/mattt/AnyDecisionModel-Examples)\n- [AnyLanguageModel on GitHub](https://github.com/huggingface/AnyLanguageModel)\n- [swift-huggingface on GitHub](https://github.com/huggingface/swift-huggingface)\n\n*Thanks to NVIDIA for Parakeet, Sortformer, and NeMo, to Moondream for Parakeet Ultra, to FluidInference, whose FluidAudio and mobius projects documented the conventions that the Parakeet runtime and the Core ML conversions follow, to Cloudflare for Clef, and to everyone who contributed to AnyLanguageModel, whose patterns both packages borrow.*\n", "authors": [], "captured_at": "2026-10-29T15:30:03.869733718+00:00", "category": "ai", "content_html": "

Language models can do a lot. They write, summarize, translate, and call tools. But they're only a small part of what machine learning can do.

\n

Say a customer calls your support line. First, you want to know what they said (speech recognition). Then, you want to decide where to send them (decision). You could hand both to a language model, but there are models built for each that are better, faster, and less expensive.

\n

In our AnyLanguageModel 1.0 recap, we looked at a year of local and hosted models in Swift. With AnyLanguageModel and swift-huggingface now on their 1.x releases, we're extending the same approach to speech and decisions: familiar Swift APIs with a choice of model.

\n

Today we're announcing two packages:

\n\n

Both packages can download open models from the Hugging Face Hub and run them on device.

\n

AnySpeechRecognizer

\n

Computers have been transcribing speech for decades, and in the past few years, they've gotten very good at it. Apple platforms already have an API for it. macOS 26 and iOS 26 introduced SpeechAnalyzer, Apple's new speech-to-text API. You add modules such as SpeechTranscriber and SpeechDetector to an analyzer, give it a file or a stream of audio buffers, and read results from async sequences as they arrive. AssetInventory installs the models, so your app decides when downloads happen.

\n

It's a well-designed API, but the only engine behind it is Apple's. AnySpeechRecognizer keeps the API and lets you choose the engine:

\n
- import Speech\n+ import AnySpeechRecognizer\n
\n

The package mirrors SpeechAnalyzer, SpeechTranscriber, DictationTranscriber, SpeechDetector, and AssetInventory, with Apple's Speech framework as the default engine. If your code uses that shared API, changing the import is enough. To use Parakeet instead, change only how you create the transcriber:

\n
let transcriber = SpeechTranscriber(\n    model: ParakeetSpeechModel(variant: .v3),\n    transcriptionOptions: [],\n    reportingOptions: [.volatileResults],\n    attributeOptions: [.audioTimeRange]\n)\n
\n

The steps for installing assets, running the analyzer, and reading results stay the same. The README has a complete example and lists the differences from Apple's framework.

\n

Meet the models

\n

Parakeet v3 is NVIDIA's Parakeet TDT 0.6B v3. It's a polyglot: it recognizes 25 European languages and works out which one it's hearing, so you don't pass a locale.

\n

Parakeet Ultra is Moondream's post-trained take on v3. It was more accurate in most of our measurements.

\n

Parakeet Unified is NVIDIA's English-only Parakeet.

\n

All three Parakeets run on the package's own Core ML runtime, which supports file and live input, volatile results, word timing, and word confidence.

\n

Qwen3-ASR is Alibaba's speech model, in 0.6B and 1.7B sizes. It recognizes 30 languages and 22 Chinese dialects, including Mandarin, Cantonese, Japanese, and Korean, which Parakeet doesn't cover. It detects the language automatically, or you can set one:

\n
let transcriber = SpeechTranscriber(\n    model: QwenSpeechModel(variant: .large, language: Locale.Language(identifier: "yue")),\n    transcriptionOptions: [],\n    reportingOptions: [],\n    attributeOptions: []\n)\n
\n

Qwen3-ASR gives one final result for each segment of up to 40 seconds, without volatile results, word timing, or confidence. It runs on the GPU and is slower than Parakeet, so it's better suited to transcribing files than live input.

\n

To add another engine, implement SpeechRecognitionModel for its configuration and SpeechRecognitionExecutor for one module's session. The README includes a small provider that reports placeholder text to help you get started.

\n

Core ML models on the Hub

\n

To run these models in apps, we converted them to Core ML and published them on the Hub:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ModelCore ML conversionOriginal checkpointLicense
Parakeet TDT 0.6B v3mattt/parakeet-tdt-0.6b-v3-coremlnvidia/parakeet-tdt-0.6b-v3CC BY 4.0
Parakeet Ultramattt/parakeet-ultra-coremlmoondream/parakeet-ultraCC BY 4.0
Parakeet Unified EN 0.6Bmattt/parakeet-unified-en-0.6b-coremlnvidia/parakeet-unified-en-0.6bNVIDIA Open Model License
Qwen3-ASR 0.6Bmattt/qwen3-asr-0.6b-coremlQwen/Qwen3-ASR-0.6B-hfApache 2.0
Qwen3-ASR 1.7Bmattt/qwen3-asr-1.7b-coremlQwen/Qwen3-ASR-1.7B-hfApache 2.0
Streaming Sortformer 4spk v2mattt/diar-streaming-sortformer-4spk-v2-coremlnvidia/diar_streaming_sortformer_4spk-v2CC BY 4.0
Streaming Sortformer 4spk v2.1mattt/diar-streaming-sortformer-4spk-v2.1-coremlnvidia/diar_streaming_sortformer_4spk-v2.1NVIDIA Open Model License
\n

Each repository records its source checkpoint and a checksum for every file, and the package pins it to a fixed revision. The Parakeet repositories add an 8-second encoder for live input, which cuts live analysis time by about a third at the cost of a somewhat higher word error rate. The repository also has the conversion workflows, most of which follow FluidInference's mobius scripts, and checks each conversion against the original model.

\n

Downloads happen only when your app asks for them: about 600 MB for each Parakeet, plus about 590 MB for the live encoder, 0.9 to 2.4 GB for Qwen3-ASR, and about 250 MB for Sortformer. Files go in the standard Hugging Face cache, so a Mac app outside the sandbox shares them with Python and other Hugging Face tools.

\n

Speaker diarization

\n

When you transcribe a conversation, you often want to know who said what. That's speaker diarization, and Apple's Speech framework doesn't have a module for it, so AnySpeechRecognizer adds one. SpeakerDiarizer runs NVIDIA's Streaming Sortformer on Core ML alongside an Apple or Parakeet transcriber, on the same timeline:

\n
let diarizer = SpeakerDiarizer(model: SortformerDiarizationModel())\nif let request = try await AssetInventory.assetInstallationRequest(\n    supporting: [transcriber, diarizer]\n) {\n    try await request.downloadAndInstall()\n}\nlet analyzer = SpeechAnalyzer(modules: [transcriber, diarizer])\n
\n

The diarizer tracks up to four speakers, even when they talk over each other. A SpeakerTimeline collects its results and attributes each timed word to a speaker, so you can print a transcript by speaker turn. Like image support in AnyLanguageModel, this goes beyond what Apple's API offers today.

\n

The diarizer uses Sortformer v2 by default. You can also choose v2.1, which NVIDIA trained with more meeting data and which was more accurate on meetings in our measurements, too.

\n

How well does it work?

\n

We measured the package with its hear command-line tool on an M4 Max Mac Studio with macOS 27.0.1. Here are the word error rates on LibriSpeech files of 3 to 8 minutes:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Transcribertest-cleantest-other
Parakeet v3, file1.55%2.78%
Parakeet v3, live4.73%4.29%
Parakeet Ultra, file1.66%2.55%
Parakeet Ultra, live2.65%3.28%
Parakeet Unified, file1.43%2.19%
Parakeet Unified, live1.45%2.42%
Apple's transcriber, file2.12%3.97%
\n

Parakeet transcribes an hour of audio in about 20 to 40 seconds from a file, or about 1.5 to 2.5 minutes live, with about 100 MB of memory once Core ML has compiled the model. Qwen3-ASR is much slower and needs a few gigabytes of memory. Meetings are hard for everyone: on AMI recordings, where people talk over each other, every engine we tested had a word error rate above 23%.

\n

These numbers come from one machine and a few datasets, so measure with your own hardware and audio. hear evaluate works on your own files, and the repository's natural-speech-eval.py tool reproduces these measurements. So far, we've tested only on macOS 27, not on an iPhone.

\n

Next, we'd like to add more engines, including Core AI on macOS 27 and iOS 27.

\n

AnyDecisionModel

\n

AnyDecisionModel is a different story. There's no Apple framework to map onto, and decision models aren't new, exactly.

\n
\n

It's only a decision model if it comes from the Décision region of France; otherwise, it's sparkling zero-shot classification.

\n
\n

What's new is the name, a lot of hype, and a crop of models trained to make decisions. We think that's where things are going, so we wanted a Swift API that's ready for them.

\n

Apps often ask language models to make decisions. Is this message urgent? Which team should handle this ticket? How frustrated is this customer? The usual approach is to prompt a language model and parse the text that it generates. That gives you one answer, but doesn't show the probabilities it assigned to the alternatives.

\n

Here's what a decision looks like in Swift:

\n
import AnyDecisionModel\n\nenum Department: String, Choosable {\n    case billing, technical, sales\n}\n\nlet ticket = "I was charged twice for my order, and now the app logs me out when I try to ask for a refund."\n\nlet model = MLXDecisionModel(modelID: "mlx-community/Qwen3-4B-Instruct-2507-4bit")\nlet session = DecisionSession(model: model, state: .text(ticket))\n\nlet urgent = try await session.probability(of: "Does this convey urgency?")\nlet team = try await session.choice("Which team?", from: Department.self)\nlet anger = try await session.score(\n    "How frustrated?",\n    levels: ["Calm", "Frustrated", "Very angry"]\n)\n
\n

urgent is a Double. team is a Choice<Department> containing the selected case, a probability for each option, and a confidence value. anger is a Score containing the expected level, a probability for each level, and a confidence value. Because Department conforms to Choosable, team.value is a Department that you can use in an exhaustive switch.

\n

If you've used Foundation Models or AnyLanguageModel, the model-and-session pattern will look familiar. A DecisionSession holds a single state, which can be text or JSON, and every question is about that state. Questions are independent: an answer never becomes input to a later question. To ask several questions at once, pass them to decide(_:) as a batch.

\n

For each question, the MLX backend builds a prompt with a one-token label for each allowed answer, then reads the probability of each label from one forward pass.

\n

Try it out: Emoji Magnet

\n

Emoji Magnet is a macOS app with a text field above a pile of 343 emoji. Type a phrase, such as "things you wear", and the emoji that match rise out of the pile. Refine it to "things you wear in winter", and the emoji that no longer match fall back.

\n

The app has no built-in keyword rules; the model decides everything at inference time. When you pause typing, it asks the model one yes-or-no question for each emoji, for example:

\n
Statement: a heavy winter coat fits the description "things you wear". Is this statement true?\n
\n

The session's state holds the phrase. The demo above uses the hosted Jev backend, which answers all 343 questions in one request, so the emoji settle in about half a second for each phrase. Without a TYPESAFE_AI_API_KEY, the app runs Qwen3 4B on your Mac with the MLX backend. That backend uses prefix caching to evaluate the state once, then answers 16 questions per forward pass. On an M4 Max, a group of 16 takes about 0.45 seconds after the model loads. A full pass over all 343 emoji takes about 10 seconds, and each group's results appear as soon as they arrive.

\n

Emoji Magnet is in the AnyDecisionModel-Examples repository.

\n

Decision heads and Clef

\n

Reading next-token probabilities works with general-purpose language models, but those models weren't trained to make decisions. Cloudflare's Clef models are an example of a newer approach: a model with a trained decision head that turns a state and a schema of typed questions into decisions, returning probabilities for all allowed options across the questions in one forward pass. Clef-Flash is the 9B variant.

\n

AnyDecisionModel runs Clef-Flash on a Mac with MLX:

\n
let model = MLXHeadDecisionModel(\n    modelID: "mlx-community/clef-flash-4bit",\n    head: .jointSchema\n)\n
\n

You use the same sessions and questions, but MLXHeadDecisionModel reads each decision from Clef's trained decision head instead of from next-token probabilities. The 4-bit model is a 6.2 GB download, and its model card recommends a Mac with at least 16 GB of memory. Clef can read images and video, but this backend accepts only text and JSON states.

\n

Calibrating and checking probabilities

\n

Raw probabilities from language models are often overconfident. The local backends accept a Calibration, which applies a temperature to every answer and a bias to yes-or-no answers. Use separate datasets to fit the calibration and evaluate the results.

\n

Language models can also favor an option because of its position in a list. On the MLX, Core AI, and llama.cpp next-token backends, turn on rotationDebiasing to ask a choice question once for each rotation of the options and average the probabilities. That costs one prompt evaluation per rotation.

\n

Each local choice and score also includes a confidence value based on the entropy of its probabilities. It measures how concentrated the probabilities are, not whether the answer is right, so check the error rate on your own data before you pick a threshold. The next-token backends' diagnostics report how much probability the model assigned to the allowed answers. A low value means it assigned most of the probability elsewhere.

\n

Backends

\n

AnyDecisionModel has six backends:

\n\n

The next-token backends label answers "yes" and "no", A to Z, or 0 to 9, so a choice can have at most 26 options, and a score at most 10 levels. Decision heads have no such limit.

\n

Like AnyLanguageModel, the package uses Swift package traits. Without traits, it has no third-party dependencies. You get the core types and the Jev and Ollama backends, which also build on Linux. Enable the MLX, CoreAI, or Llama trait for the local backends.

\n

Get started

\n

Both packages require Swift 6.2 or later and macOS 26 or iOS 26 or later. The Parakeet and Sortformer runtimes and the MLX backend require Apple silicon. AnyDecisionModel's MLX build tasks use Xcode's Swift 6.4 or later and the Metal toolchain; Core AI requires Xcode 27 and an OS 27 runtime.

\n

Add the packages to your Package.swift:

\n
dependencies: [\n    .package(url: "https://github.com/mattt/AnySpeechRecognizer", from: "0.1.0"),\n    .package(\n        url: "https://github.com/mattt/AnyDecisionModel",\n        from: "0.6.0",\n        traits: ["MLX"]\n    ),\n]\n
\n

For now, AnyDecisionModel's CoreAI trait can't be combined with AnySpeechRecognizer, because coreai-kit 0.7.3 requires swift-huggingface below 1.0 and AnySpeechRecognizer requires 1.x.

\n

Each README has more examples and details about requirements and backends. AnySpeechRecognizer also includes a command-line tool, hear, that transcribes local files, labels their speakers, and evaluates the results. Every hear command requires --backend apple, --backend parakeet, or --backend qwen; run hear prepare with the same backend and model options before analysis to install the needed assets.

\n

Try them in your apps and tell us what works and what doesn't. Issues and pull requests are welcome on both repositories.

\n

Links

\n\n

Thanks to NVIDIA for Parakeet, Sortformer, and NeMo, to Moondream for Parakeet Ultra, to FluidInference, whose FluidAudio and mobius projects documented the conventions that the Parakeet runtime and the Core ML conversions follow, to Cloudflare for Clef, and to everyone who contributed to AnyLanguageModel, whose patterns both packages borrow.

\n", "content_markdown": "Language models can do a lot. They write, summarize, translate, and call tools. But they're only a small part of what machine learning can do.\n\nSay a customer calls your support line. First, you want to know what they said (speech recognition). Then, you want to decide where to send them (decision). You *could* hand both to a language model, but there are models built for each that are better, faster, and less expensive.\n\nIn our [AnyLanguageModel 1.0 recap](https://huggingface.co/blog/anylanguagemodel-1-0), we looked at a year of local and hosted models in Swift. With AnyLanguageModel and swift-huggingface now on their 1.x releases, we're extending the same approach to speech and decisions: familiar Swift APIs with a choice of model.\n\nToday we're announcing two packages:\n\n- [AnySpeechRecognizer](https://github.com/mattt/AnySpeechRecognizer) implements Apple's new speech-to-text API with a choice of recognition engine, including NVIDIA's Parakeet and Alibaba's Qwen3-ASR running locally on Core ML, and adds speaker diarization to track who speaks when.\n- [AnyDecisionModel](https://github.com/mattt/AnyDecisionModel) answers yes-or-no, multiple-choice, and rating-scale questions with typed results and probabilities. It derives each answer from the model's probabilities instead of generating text, using a model on your device with MLX, Core AI, or llama.cpp, or a decision model from Jev or Ollama.\n\nBoth packages can download open models from the Hugging Face Hub and run them on device.\n\n## AnySpeechRecognizer\n\nComputers have been transcribing speech for decades, and in the past few years, they've gotten very good at it. Apple platforms already have an API for it. macOS 26 and iOS 26 introduced `SpeechAnalyzer`, Apple's new speech-to-text API. You add modules such as `SpeechTranscriber` and `SpeechDetector` to an analyzer, give it a file or a stream of audio buffers, and read results from async sequences as they arrive. `AssetInventory` installs the models, so your app decides when downloads happen.\n\nIt's a well-designed API, but the only engine behind it is Apple's. AnySpeechRecognizer keeps the API and lets you choose the engine:\n\n```diff\n- import Speech\n+ import AnySpeechRecognizer\n```\n\nThe package mirrors `SpeechAnalyzer`, `SpeechTranscriber`, `DictationTranscriber`, `SpeechDetector`, and `AssetInventory`, with Apple's Speech framework as the default engine. If your code uses that shared API, changing the import is enough. To use Parakeet instead, change only how you create the transcriber:\n\n```swift\nlet transcriber = SpeechTranscriber(\n model: ParakeetSpeechModel(variant: .v3),\n transcriptionOptions: [],\n reportingOptions: [.volatileResults],\n attributeOptions: [.audioTimeRange]\n)\n```\n\nThe steps for installing assets, running the analyzer, and reading results stay the same. The [README](https://github.com/mattt/AnySpeechRecognizer#transcribe-a-file) has a complete example and [lists the differences](https://github.com/mattt/AnySpeechRecognizer#differences-from-apples-speech-framework) from Apple's framework.\n\n### Meet the models\n\n**Parakeet v3** is NVIDIA's [Parakeet TDT 0.6B v3](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3). It's a polyglot: it recognizes 25 European languages and works out which one it's hearing, so you don't pass a locale.\n\n**Parakeet Ultra** is Moondream's [post-trained take on v3](https://huggingface.co/moondream/parakeet-ultra). It was more accurate in most of our measurements.\n\n**Parakeet Unified** is NVIDIA's [English-only Parakeet](https://huggingface.co/nvidia/parakeet-unified-en-0.6b).\n\nAll three Parakeets run on the package's own Core ML runtime, which supports file and live input, volatile results, word timing, and word confidence.\n\n**Qwen3-ASR** is Alibaba's speech model, in [0.6B](https://huggingface.co/Qwen/Qwen3-ASR-0.6B-hf) and [1.7B](https://huggingface.co/Qwen/Qwen3-ASR-1.7B-hf) sizes. It recognizes 30 languages and 22 Chinese dialects, including Mandarin, Cantonese, Japanese, and Korean, which Parakeet doesn't cover. It detects the language automatically, or you can set one:\n\n```swift\nlet transcriber = SpeechTranscriber(\n model: QwenSpeechModel(variant: .large, language: Locale.Language(identifier: \"yue\")),\n transcriptionOptions: [],\n reportingOptions: [],\n attributeOptions: []\n)\n```\n\nQwen3-ASR gives one final result for each segment of up to 40 seconds, without volatile results, word timing, or confidence. It runs on the GPU and is slower than Parakeet, so it's better suited to transcribing files than live input.\n\nTo add another engine, implement `SpeechRecognitionModel` for its configuration and `SpeechRecognitionExecutor` for one module's session. The README includes a [small provider](https://github.com/mattt/AnySpeechRecognizer#custom-providers) that reports placeholder text to help you get started.\n\n### Core ML models on the Hub\n\nTo run these models in apps, we converted them to Core ML and published them on the Hub:\n\n| Model | Core ML conversion | Original checkpoint | License |\n| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |\n| Parakeet TDT 0.6B v3 | [mattt/parakeet-tdt-0.6b-v3-coreml](https://huggingface.co/mattt/parakeet-tdt-0.6b-v3-coreml) | [nvidia/parakeet-tdt-0.6b-v3](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3) | CC BY 4.0 |\n| Parakeet Ultra | [mattt/parakeet-ultra-coreml](https://huggingface.co/mattt/parakeet-ultra-coreml) | [moondream/parakeet-ultra](https://huggingface.co/moondream/parakeet-ultra) | CC BY 4.0 |\n| Parakeet Unified EN 0.6B | [mattt/parakeet-unified-en-0.6b-coreml](https://huggingface.co/mattt/parakeet-unified-en-0.6b-coreml) | [nvidia/parakeet-unified-en-0.6b](https://huggingface.co/nvidia/parakeet-unified-en-0.6b) | [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/) |\n| Qwen3-ASR 0.6B | [mattt/qwen3-asr-0.6b-coreml](https://huggingface.co/mattt/qwen3-asr-0.6b-coreml) | [Qwen/Qwen3-ASR-0.6B-hf](https://huggingface.co/Qwen/Qwen3-ASR-0.6B-hf) | Apache 2.0 |\n| Qwen3-ASR 1.7B | [mattt/qwen3-asr-1.7b-coreml](https://huggingface.co/mattt/qwen3-asr-1.7b-coreml) | [Qwen/Qwen3-ASR-1.7B-hf](https://huggingface.co/Qwen/Qwen3-ASR-1.7B-hf) | Apache 2.0 |\n| Streaming Sortformer 4spk v2 | [mattt/diar-streaming-sortformer-4spk-v2-coreml](https://huggingface.co/mattt/diar-streaming-sortformer-4spk-v2-coreml) | [nvidia/diar\\_streaming\\_sortformer\\_4spk-v2](https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2) | CC BY 4.0 |\n| Streaming Sortformer 4spk v2.1 | [mattt/diar-streaming-sortformer-4spk-v2.1-coreml](https://huggingface.co/mattt/diar-streaming-sortformer-4spk-v2.1-coreml) | [nvidia/diar\\_streaming\\_sortformer\\_4spk-v2.1](https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2.1) | [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/) |\n\nEach repository records its source checkpoint and a checksum for every file, and the package pins it to a fixed revision. The Parakeet repositories add an 8-second encoder for live input, which cuts live analysis time by about a third at the cost of a somewhat higher word error rate. The repository also has the conversion workflows, most of which follow FluidInference's [mobius](https://github.com/FluidInference/mobius) scripts, and checks each conversion against the original model.\n\nDownloads happen only when your app asks for them: about 600 MB for each Parakeet, plus about 590 MB for the live encoder, 0.9 to 2.4 GB for Qwen3-ASR, and about 250 MB for Sortformer. Files go in the standard Hugging Face cache, so a Mac app outside the sandbox shares them with Python and other Hugging Face tools.\n\n### Speaker diarization\n\nWhen you transcribe a conversation, you often want to know who said what. That's speaker diarization, and Apple's Speech framework doesn't have a module for it, so AnySpeechRecognizer adds one. `SpeakerDiarizer` runs NVIDIA's [Streaming Sortformer](https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2) on Core ML alongside an Apple or Parakeet transcriber, on the same timeline:\n\n```swift\nlet diarizer = SpeakerDiarizer(model: SortformerDiarizationModel())\nif let request = try await AssetInventory.assetInstallationRequest(\n supporting: [transcriber, diarizer]\n) {\n try await request.downloadAndInstall()\n}\nlet analyzer = SpeechAnalyzer(modules: [transcriber, diarizer])\n```\n\nThe diarizer tracks up to four speakers, even when they talk over each other. A `SpeakerTimeline` collects its results and attributes each timed word to a speaker, so you can print a transcript by speaker turn. Like image support in AnyLanguageModel, this goes beyond what Apple's API offers today.\n\nThe diarizer uses Sortformer v2 by default. You can also choose [v2.1](https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2.1), which NVIDIA trained with more meeting data and which was more accurate on meetings in our measurements, too.\n\n### How well does it work?\n\nWe measured the package with its `hear` command-line tool on an M4 Max Mac Studio with macOS 27.0.1. Here are the word error rates on LibriSpeech files of 3 to 8 minutes:\n\n| Transcriber | test-clean | test-other |\n| ------------------------- | ---------- | ---------- |\n| Parakeet v3, file | 1.55% | 2.78% |\n| Parakeet v3, live | 4.73% | 4.29% |\n| Parakeet Ultra, file | 1.66% | 2.55% |\n| Parakeet Ultra, live | 2.65% | 3.28% |\n| Parakeet Unified, file | 1.43% | 2.19% |\n| Parakeet Unified, live | 1.45% | 2.42% |\n| Apple's transcriber, file | 2.12% | 3.97% |\n\nParakeet transcribes an hour of audio in about 20 to 40 seconds from a file, or about 1.5 to 2.5 minutes live, with about 100 MB of memory once Core ML has compiled the model. Qwen3-ASR is much slower and needs a few gigabytes of memory. Meetings are hard for everyone: on AMI recordings, where people talk over each other, every engine we tested had a word error rate above 23%.\n\nThese numbers come from one machine and a few datasets, so measure with your own hardware and audio. `hear evaluate` works on your own files, and the repository's `natural-speech-eval.py` tool [reproduces these measurements](https://github.com/mattt/AnySpeechRecognizer#accuracy-and-performance). So far, we've tested only on macOS 27, not on an iPhone.\n\nNext, we'd like to add more engines, including Core AI on macOS 27 and iOS 27.\n\n## AnyDecisionModel\n\nAnyDecisionModel is a different story. There's no Apple framework to map onto, and decision models aren't new, exactly.\n\n> It's only a decision model if it comes from the Décision region of France; otherwise, it's sparkling zero-shot classification.\n\nWhat's new is the name, a lot of hype, and a crop of models trained to make decisions. We think that's where things are going, so we wanted a Swift API that's ready for them.\n\nApps often ask language models to make decisions. *Is this message urgent?* *Which team should handle this ticket?* *How frustrated is this customer?* The usual approach is to prompt a language model and parse the text that it generates. That gives you one answer, but doesn't show the probabilities it assigned to the alternatives.\n\nHere's what a decision looks like in Swift:\n\n```swift\nimport AnyDecisionModel\n\nenum Department: String, Choosable {\n case billing, technical, sales\n}\n\nlet ticket = \"I was charged twice for my order, and now the app logs me out when I try to ask for a refund.\"\n\nlet model = MLXDecisionModel(modelID: \"mlx-community/Qwen3-4B-Instruct-2507-4bit\")\nlet session = DecisionSession(model: model, state: .text(ticket))\n\nlet urgent = try await session.probability(of: \"Does this convey urgency?\")\nlet team = try await session.choice(\"Which team?\", from: Department.self)\nlet anger = try await session.score(\n \"How frustrated?\",\n levels: [\"Calm\", \"Frustrated\", \"Very angry\"]\n)\n```\n\n`urgent` is a `Double`. `team` is a `Choice` containing the selected case, a probability for each option, and a confidence value. `anger` is a `Score` containing the expected level, a probability for each level, and a confidence value. Because `Department` conforms to `Choosable`, `team.value` is a `Department` that you can use in an exhaustive `switch`.\n\nIf you've used Foundation Models or AnyLanguageModel, the model-and-session pattern will look familiar. A `DecisionSession` holds a single state, which can be text or JSON, and every question is about that state. Questions are independent: an answer never becomes input to a later question. To ask several questions at once, pass them to `decide(_:)` as a batch.\n\nFor each question, the MLX backend builds a prompt with a one-token label for each allowed answer, then reads the probability of each label from one forward pass.\n\n### Try it out: Emoji Magnet\n\nEmoji Magnet is a macOS app with a text field above a pile of 343 emoji. Type a phrase, such as \"things you wear\", and the emoji that match rise out of the pile. Refine it to \"things you wear in winter\", and the emoji that no longer match fall back.\n\nThe app has no built-in keyword rules; the model decides everything at inference time. When you pause typing, it asks the model one yes-or-no question for each emoji, for example:\n\n```\nStatement: a heavy winter coat fits the description \"things you wear\". Is this statement true?\n```\n\nThe session's state holds the phrase. The demo above uses the hosted Jev backend, which answers all 343 questions in one request, so the emoji settle in about half a second for each phrase. Without a `TYPESAFE_AI_API_KEY`, the app runs Qwen3 4B on your Mac with the MLX backend. That backend uses prefix caching to evaluate the state once, then answers 16 questions per forward pass. On an M4 Max, a group of 16 takes about 0.45 seconds after the model loads. A full pass over all 343 emoji takes about 10 seconds, and each group's results appear as soon as they arrive.\n\nEmoji Magnet is in the [AnyDecisionModel-Examples](https://github.com/mattt/AnyDecisionModel-Examples) repository.\n\n### Decision heads and Clef\n\nReading next-token probabilities works with general-purpose language models, but those models weren't trained to make decisions. Cloudflare's [Clef](https://huggingface.co/Cloudflare/clef) models are an example of a newer approach: a model with a trained decision head that turns a state and a schema of typed questions into decisions, returning probabilities for all allowed options across the questions in one forward pass. [Clef-Flash](https://huggingface.co/Cloudflare/clef-flash) is the 9B variant.\n\nAnyDecisionModel runs Clef-Flash on a Mac with MLX:\n\n```swift\nlet model = MLXHeadDecisionModel(\n modelID: \"mlx-community/clef-flash-4bit\",\n head: .jointSchema\n)\n```\n\nYou use the same sessions and questions, but `MLXHeadDecisionModel` reads each decision from Clef's trained decision head instead of from next-token probabilities. The [4-bit model](https://huggingface.co/mlx-community/clef-flash-4bit) is a 6.2 GB download, and its model card recommends a Mac with at least 16 GB of memory. Clef can read images and video, but this backend accepts only text and JSON states.\n\n### Calibrating and checking probabilities\n\nRaw probabilities from language models are often overconfident. The local backends accept a `Calibration`, which applies a temperature to every answer and a bias to yes-or-no answers. Use separate datasets to fit the calibration and evaluate the results.\n\nLanguage models can also favor an option because of its position in a list. On the MLX, Core AI, and llama.cpp next-token backends, turn on `rotationDebiasing` to ask a choice question once for each rotation of the options and average the probabilities. That costs one prompt evaluation per rotation.\n\nEach local choice and score also includes a confidence value based on the entropy of its probabilities. It measures how concentrated the probabilities are, not whether the answer is right, so check the error rate on your own data before you pick a threshold. The next-token backends' diagnostics report how much probability the model assigned to the allowed answers. A low value means it assigned most of the probability elsewhere.\n\n### Backends\n\nAnyDecisionModel has six backends:\n\n- **MLX** runs a small language model on Apple silicon with [mlx-swift-lm](https://github.com/ml-explore/mlx-swift-lm) and downloads it from the Hub.\n- **MLX decision heads** run a model with its own trained decision head, such as Clef.\n- **Core AI** does the same as MLX with a Core AI model on macOS 27 and iOS 27.\n- **llama.cpp** reads decisions from a local GGUF model through [llama.swift](https://github.com/mattt/llama.swift).\n- **Jev** calls TypeSafe's Jev through the [System One API](https://docs.typesafe.ai/api/system-one), or any server that implements it, such as [Kev](https://github.com/jaredpalmer/kev).\n- **Ollama** calls a System One decision model on a local Ollama server or in Ollama's cloud. It requires Ollama 0.35.0 or later, or 0.40.5 or later for cloud models.\n\nThe next-token backends label answers \"yes\" and \"no\", A to Z, or 0 to 9, so a choice can have at most 26 options, and a score at most 10 levels. Decision heads have no such limit.\n\nLike AnyLanguageModel, the package uses Swift package traits. Without traits, it has no third-party dependencies. You get the core types and the Jev and Ollama backends, which also build on Linux. Enable the `MLX`, `CoreAI`, or `Llama` trait for the local backends.\n\n## Get started\n\nBoth packages require Swift 6.2 or later and macOS 26 or iOS 26 or later. The Parakeet and Sortformer runtimes and the MLX backend require Apple silicon. AnyDecisionModel's MLX build tasks use Xcode's Swift 6.4 or later and the Metal toolchain; Core AI requires Xcode 27 and an OS 27 runtime.\n\nAdd the packages to your `Package.swift`:\n\n```swift\ndependencies: [\n .package(url: \"https://github.com/mattt/AnySpeechRecognizer\", from: \"0.1.0\"),\n .package(\n url: \"https://github.com/mattt/AnyDecisionModel\",\n from: \"0.6.0\",\n traits: [\"MLX\"]\n ),\n]\n```\n\nFor now, AnyDecisionModel's `CoreAI` trait can't be combined with AnySpeechRecognizer, because coreai-kit 0.7.3 requires swift-huggingface below 1.0 and AnySpeechRecognizer requires 1.x.\n\nEach README has more examples and details about requirements and backends. AnySpeechRecognizer also includes a command-line tool, `hear`, that transcribes local files, labels their speakers, and evaluates the results. Every `hear` command requires `--backend apple`, `--backend parakeet`, or `--backend qwen`; run `hear prepare` with the same backend and model options before analysis to install the needed assets.\n\nTry them in your apps and tell us what works and what doesn't. Issues and pull requests are welcome on both repositories.\n\n## Links\n\n- [AnySpeechRecognizer on GitHub](https://github.com/mattt/AnySpeechRecognizer)\n- [AnyDecisionModel on GitHub](https://github.com/mattt/AnyDecisionModel)\n- [AnyDecisionModel-Examples on GitHub](https://github.com/mattt/AnyDecisionModel-Examples)\n- [AnyLanguageModel on GitHub](https://github.com/huggingface/AnyLanguageModel)\n- [swift-huggingface on GitHub](https://github.com/huggingface/swift-huggingface)\n\n*Thanks to NVIDIA for Parakeet, Sortformer, and NeMo, to Moondream for Parakeet Ultra, to FluidInference, whose FluidAudio and mobius projects documented the conventions that the Parakeet runtime and the Core ML conversions follow, to Cloudflare for Clef, and to everyone who contributed to AnyLanguageModel, whose patterns both packages borrow.*\n", "dateCreated": "2026-10-29T15:30:03.869733718+00:00", "date_created": "2026-10-29T00:00:00+00:00", "date_modified": "2026-10-29T00:00:00+00:00", "date_published": "2026-10-29T00:00:00+00:00", "description": "Language models can do a lot. They write, summarize, translate, and call tools. But they're only a small part of what machine learning can do. Say a customer calls your support line. First, you want to know what they said (speech…", "external_url": "https://huggingface.co/blog/anydecisionmodel", "git": { "commit": "20068632b65d2830a2d54f25ca1d90d697bea2c0", "path": "items/huggingface-co/2026/10/2026-10-29-introducing-anyspeechrecognizer-and-anydecisionmodel-open.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/20068632b65d2830a2d54f25ca1d90d697bea2c0/items/huggingface-co/2026/10/2026-10-29-introducing-anyspeechrecognizer-and-anydecisionmodel-open.md" }, "id": "urn:aggr:item:1a4570cd52059f09582e1b00107c8a091ba41153", "isBasedOn": { "@id": "https://huggingface.co/blog/anydecisionmodel" }, "mainEntity": { "@id": "https://huggingface.co/blog/anydecisionmodel", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-10-29-introducing-anyspeechrecognizer-and-anydecisionmodel-open/#webpage" }, "dateModified": "2026-10-29T00:00:00+00:00", "datePublished": "2026-10-29T00:00:00+00:00", "headline": "Introducing AnySpeechRecognizer and AnyDecisionModel: Open Models for Speech and Decisions on Apple Platforms", "image": { "@type": "ImageObject", "caption": null, "height": 90, "url": "https://huggingface.co/blog/assets/anydecisionmodel/banner.png", "width": 160 }, "timeRequired": "PT12M", "url": "https://huggingface.co/blog/anydecisionmodel", "wordCount": 2598 }, "name": "Archived snapshot: Introducing AnySpeechRecognizer and AnyDecisionModel: Open Models for Speech and Decisions on Apple Platforms", "preview": { "alt": null, "color": null, "height": 90, "placeholder": { "data_url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII=", "hash": "" }, "url": "https://huggingface.co/blog/assets/anydecisionmodel/banner.png", "width": 160 }, "reading_minutes": 12, "replicated_at": null, "source": "huggingface-co", "source_name": "Hugging Face - Blog", "summary": null, "tags": [], "temporalCoverage": "2026-10-29T15:30:03.869733718+00:00", "title": "Introducing AnySpeechRecognizer and AnyDecisionModel: Open Models for Speech and Decisions on Apple Platforms", "url": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-10-29-introducing-anyspeechrecognizer-and-anydecisionmodel-open/", "word_count": 2598 }