With my kids I’m currently building a little dungeron crawler and we’re trying to learn a bit about game development along the way. However given that we live in the modern times of increasing capabilities we wanted to get a narrator and voice over.
\nBut we also wanted it to not sound like everyone’s standard AI voices so I looked into what options there are on the table. After some investigation I narrowed down to using Qwen3-TTS which uses voice embeddings for custom voices.
\nWith a bit of creativity you can also do more with those custom voices than appears initially. In case this is of interest to you, this post just covers what we learned and with some examples. You can probably point your agent to it and it can do something with it.
\nQwen3 is not the only TTS model in town, and it wans’t even my first choice. We initially used Kokoro. However Qwen3 supports voice cloning and we wanted to have it feel a bit more unique.
\nIn practice it’s a bit tricker though. Qwen3 ships three kinds of models:
\n1.7B-Base (and 0.6B): clones a voice from a few seconds of reference audio1.7B-CustomVoice: comes with nine built-in speakers and instructions1.7B-VoiceDesign: invents a voice from a text descriptionThe base model clones either in context or from the speaker embedding alone but there is no documented way to clone a voice and direct it with instructions. That means unlike the custom voice and voice design models you cannot say “speak sad” or “whisper” with a cloned base model.
\nAt present the official options for a custom voice are fine-tuning Base on a single speaker or to make a clip with VoiceDesign and clone it with Base. That is not all that bad, but it means instructions are gone.
\nFor this blog post I took 28 seconds of me from my talk at PyAI Conf, cut between two pauses so that it starts and ends on a full sentence, and transcribed it with Whisper. The transcript is really only needed for baking but to compare against an already existing voice it’s useful to have it say the same thing as the original recording.
\nNote that a conference talk recording is not ideal but it works anyways. Which is also a good reminder that you really don’t need a particularly clean recording (or a very long one) to get a halfway convincing version of a person. Particularly when then presented to you on a low quality phone call, it won’t be very hard to trick your grandmother.
\nThe first thing you can do is to blend that recording with another voice. For instance you can use VoiceDesign to make up a voice from a description. For this blog post the designed voice was created like this, with Qwen’s own Python package qwen-tts (pip install qwen-tts, I used 0.1.1), which the rest of the code in this post uses too:
import torch\nfrom qwen_tts import Qwen3TTSModel\n\nkw = dict(device_map="mps", dtype=torch.bfloat16, attn_implementation="sdpa")\ndesign = Qwen3TTSModel.from_pretrained("Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign", **kw)\nwavs, sr = design.generate_voice_design(\n text=open("voices/transcript.txt").read(), language="English",\n instruct="A man in his forties with a warm, natural, medium-low voice, speaking "\n "clearly and calmly at an easy conversational pace, like someone reading "\n "aloud to a friend.",\n)\nYou can see how the two different voices behave, particularly when blended together:
\nLoading the voice samples…
\nWhat Base takes from a recording when cloning from the embedding alone is a speaker embedding which is basically 2048 numbers out of a speaker encoder that was trained. The same voice gets similar numbers and different voices get different ones. Which btw, is also a convenient way to do speaker detection from recordings if you want to! It’s not perfect, but it’s quite helpful for a single audio track recording to tell two speakers apart if you do some basic slicing and dicing (more on that below).
\nBecause it is just a vector, you can do arithmetic on it. For instance a weighted average of two embeddings is a spaker that is somewhere between two voices:
\n = \n\n\n = \n return .\n\n\n , = , \n = * + * \n return \n\n, = , \nHere is the same sentence in eleven blends:
\nLoading the voice samples…
\nNote that it stops sounding like me quite quickly, but it retains some of my features. And it for sure ends up with creating a unique voice regardless. I found some better successes when ensuing that the two voices have similar spectograms (eg: similar “recording setups”). In this recording you can hear that one voice picked up by the lapel mic was a bit hollow compared to the designed voice.
\nIf you like a a voice that comes out, you can bake it back into a new recording that you can then use as reference:
\nimport soundfile as sf\n\nmixed = blend(armin, designed, 0.5)\naudio, rate = say(open("voices/transcript.txt").read(), mixed) # say() is shown further down\nsf.write("voices/final.flac", audio, rate)\nfinal = fingerprint("voices/final.flac")\nThe baked voice does not come back exactly as the blend but it makes handling it easier if you want to. It’s just one voice reference file.
\nSo on the question of using those embeddings for speaker recognition: how good does that work? I took an episode of State of Agentic Coding, the podcast Ben and I record, cut it into 2-second windows one second apart and compared each with an embedding of me form the conference.
\nSomewhat surprising, but maybe not too surprising using my conference recording clip as the reference this does not work at all with Qwen3. If instead I take the 20 seconds where I introduce myself at the start of the episode as the reference, it gets 98% of the rest right.
\nOn the other hand using a model that was trained for speaker verification instead does manage to pick me up from the conference recording clip. For instance ECAPA, finds me from the conference clip in 97% of the windows.
\nHere are 20 minutes of that podcast, classified by speaker with the different approaches:
\nLoading the voice samples…
\nBut the embeddings a very useful for the voice cloning! But how to give them instructions? Base and CustomVoice manipulate the speaker the same way: one vector inserted into the talker’s input sequence. In CustomVoice that vector is a row of the embedding table. In Base it is the x-vector from the speaker encoder, which only Base has. The generation code takes either (voice_clone_prompt is a regular argument of generate), and the transformer weights of the two models are nearly identical. So take the embedding with Base, then hand it to CustomVoice instead of one of its speakers:
= \n\n\n = \n = \n = \n , = \n , = \n return , \n\n, = \nPick a speaker for the slot, a model and an instruction:
\nLoading the voice samples…
\nInstructions work on cloned voices, but less reliably than on the built-in speakers, and it depends on the emotion, but honestly, it came out better than I anticipated. Definitely good enough to have some true custom voices for a voice over that don’t necessarily sound like something, someone else has in their game.
\nIf a voice is a point in a 2048-dimensional space and blending works we can maybe do some other funny math on it. The obvious one is to try to see if some math can be used to manipulate gender.
\nTo find it I had VoiceDesign create 14 different voices for men and women each. When you then take the average woman minus the average man you can then move any voice along that direction:
\n = \n = \n = / 2\n = - \n = / \n\n\n """Put the voice at position t along the gender axis, keep the rest."""\n = \n return \nHere is the result for my clone:
\nLoading the voice samples…
\nAnyways, that’s that. If you want to play around with Qwen3 TTS I hope this was of some help, and maybe you got some ideas of what to do with the voice embeddings. Use it for good, not evil.
\n\n\n", "content_markdown": "With my kids I’m currently building a little [dungeron crawler](https://github.com/mitsuhiko/dungeon-crawler) and we’re trying to [learn a bit about game development](https://mitsuhiko.github.io/dungeon-crawler/learn/) along the way. However given that we live in the modern times of increasing capabilities we wanted to get a narrator and voice over.\n\nBut we also wanted it to not sound like everyone’s standard AI voices so I looked into what options there are on the table. After some investigation I narrowed down to using [Qwen3-TTS](https://github.com/QwenLM/Qwen3-TTS) which uses voice embeddings for custom voices.\n\nWith a bit of creativity you can also do more with those custom voices than appears initially. In case this is of interest to you, this post just covers what we learned and with some examples. You can probably point your agent to it and it can do something with it.\n\n## What Qwen Offers\n\nQwen3 is not the only TTS model in town, and it wans’t even my first choice. We initially used [Kokoro](https://huggingface.co/hexgrad/Kokoro-82M). However Qwen3 supports voice cloning and we wanted to have it feel a bit more unique.\n\nIn practice it’s a bit tricker though. Qwen3 ships three kinds of models:\n\n- `1.7B-Base` (and 0.6B): clones a voice from a few seconds of reference audio\n- `1.7B-CustomVoice`: comes with nine built-in speakers and instructions\n- `1.7B-VoiceDesign`: invents a voice from a text description\n\nThe base model clones either in context or from the speaker embedding alone but there is no documented way to clone a voice and direct it with instructions. That means unlike the custom voice and voice design models you cannot say “speak sad” or “whisper” with a cloned base model.\n\nAt present the official options for a custom voice are fine-tuning Base on a single speaker or to make a clip with VoiceDesign and clone it with Base. That is not all that bad, but it means instructions are gone.\n\n## Playing With It\n\nFor this blog post I took 28 seconds of me from my talk at [PyAI Conf](https://www.youtube.com/watch?v=8RHYyRUxVrA), cut between two pauses so that it starts and ends on a full sentence, and transcribed it with Whisper. The transcript is really only needed for baking but to compare against an already existing voice it’s useful to have it say the same thing as the original recording.\n\nNote that a conference talk recording is not ideal but it works anyways. Which is also a good reminder that you really don’t need a particularly clean recording (or a very long one) to get a halfway convincing version of a person. Particularly when then presented to you on a low quality phone call, it won’t be very hard to trick your grandmother.\n\nThe first thing you can do is to blend that recording with another voice. For instance you can use VoiceDesign to make up a voice from a description. For this blog post the designed voice was created like this, with Qwen’s own Python package [`qwen-tts`](https://pypi.org/project/qwen-tts/) (`pip install qwen-tts`, I used 0.1.1), which the rest of the code in this post uses too:\n\n```\nimport torch\nfrom qwen_tts import Qwen3TTSModel\n\nkw = dict(device_map=\"mps\", dtype=torch.bfloat16, attn_implementation=\"sdpa\")\ndesign = Qwen3TTSModel.from_pretrained(\"Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign\", **kw)\nwavs, sr = design.generate_voice_design(\n text=open(\"voices/transcript.txt\").read(), language=\"English\",\n instruct=\"A man in his forties with a warm, natural, medium-low voice, speaking \"\n \"clearly and calmly at an easy conversational pace, like someone reading \"\n \"aloud to a friend.\",\n)\n```\n\nYou can see how the two different voices behave, particularly when blended together:\n\nLoading the voice samples…\n\n## Voice Embeddings\n\nWhat Base takes from a recording when cloning from the embedding alone is a speaker embedding which is basically 2048 numbers out of a speaker encoder that was trained. The same voice gets similar numbers and different voices get different ones. Which btw, is also a convenient way to do speaker detection from recordings if you want to! It’s not perfect, but it’s quite helpful for a single audio track recording to tell two speakers apart if you do some basic slicing and dicing (more on that below).\n\nBecause it is just a vector, you can do arithmetic on it. For instance a weighted average of two embeddings is a spaker that is somewhere between two voices:\n\n```\nbase = Qwen3TTSModel.from_pretrained(\"Qwen/Qwen3-TTS-12Hz-1.7B-Base\", **kw)\n\ndef fingerprint(path):\n prompt = base.create_voice_clone_prompt(ref_audio=path, x_vector_only_mode=True)\n return prompt[0].ref_spk_embedding\n\ndef blend(a, b, t):\n a, b = a.float(), b.float()\n mixed = (1 - t) * a + t * b\n return (mixed / mixed.norm() * ((1 - t) * a.norm() + t * b.norm())).to(torch.bfloat16)\n\narmin, designed = fingerprint(\"voices/armin.flac\"), fingerprint(\"voices/designed.flac\")\n```\n\nHere is the same sentence in eleven blends:\n\nLoading the voice samples…\n\nNote that it stops sounding like me quite quickly, but it retains some of my features. And it for sure ends up with creating a unique voice regardless. I found some better successes when ensuing that the two voices have similar spectograms (eg: similar “recording setups”). In this recording you can hear that one voice picked up by the lapel mic was a bit hollow compared to the designed voice.\n\nIf you like a a voice that comes out, you can bake it back into a new recording that you can then use as reference:\n\n```\nimport soundfile as sf\n\nmixed = blend(armin, designed, 0.5)\naudio, rate = say(open(\"voices/transcript.txt\").read(), mixed) # say() is shown further down\nsf.write(\"voices/final.flac\", audio, rate)\nfinal = fingerprint(\"voices/final.flac\")\n```\n\nThe baked voice does not come back exactly as the blend but it makes handling it easier if you want to. It’s just one voice reference file.\n\nSo on the question of using those embeddings for speaker recognition: how good does that work? I took an episode of [State of Agentic Coding](https://www.youtube.com/watch?v=qCpY4K9jtOg), the podcast Ben and I record, cut it into 2-second windows one second apart and compared each with an embedding of me form the conference.\n\nSomewhat surprising, but maybe not too surprising using my conference recording clip as the reference this does not work at all with Qwen3. If instead I take the 20 seconds where I introduce myself at the start of the episode as the reference, it gets 98% of the rest right.\n\nOn the other hand using a model that was trained for speaker verification instead does manage to pick me up from the conference recording clip. For instance [ECAPA](https://huggingface.co/speechbrain/spkrec-ecapa-voxceleb), finds me from the conference clip in 97% of the windows.\n\nHere are 20 minutes of that podcast, classified by speaker with the different approaches:\n\nLoading the voice samples…\n\n## Instructed Cloned Voices\n\nBut the embeddings a very useful for the voice cloning! But how to give them instructions? Base and CustomVoice manipulate the speaker the same way: one vector inserted into the talker’s input sequence. In CustomVoice that vector is a row of the embedding table. In Base it is the x-vector from the speaker encoder, which only Base has. The generation code takes either (`voice_clone_prompt` is a regular argument of `generate`), and the transformer weights of the two models are nearly identical. So take the embedding with Base, then hand it to CustomVoice instead of one of its speakers:\n\n```\ncv = Qwen3TTSModel.from_pretrained(\"Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice\", **kw)\n\ndef say(text, xvec, instruct=None, language=\"English\"):\n ids = cv._tokenize_texts([cv._build_assistant_text(text)])\n told = [cv._tokenize_texts([cv._build_instruct_text(instruct)])[0]\n if instruct else None]\n voice = {\"ref_code\": [None], \"ref_spk_embedding\": [xvec],\n \"x_vector_only_mode\": [True], \"icl_mode\": [False]}\n codes, _ = cv.model.generate(\n input_ids=ids, instruct_ids=told, voice_clone_prompt=voice,\n languages=[language], non_streaming_mode=True,\n **cv._merge_generate_kwargs(temperature=0.7, subtalker_temperature=0.7),\n )\n wavs, rate = cv.model.speech_tokenizer.decode([{\"audio_codes\": codes[0]}])\n return wavs[0], rate\n\nwav, rate = say(\"I really thought this would work.\", armin,\n \"Speak in a sad, sorrowful tone, voice low and heavy, on the verge of tears.\")\n```\n\nPick a speaker for the slot, a model and an instruction:\n\nLoading the voice samples…\n\nInstructions work on cloned voices, but less reliably than on the built-in speakers, and it depends on the emotion, but honestly, it came out better than I anticipated. Definitely good enough to have some true custom voices for a voice over that don’t necessarily sound like something, someone else has in their game.\n\n## Other Fun Math\n\nIf a voice is a point in a 2048-dimensional space and blending works we can maybe do some other funny math on it. The obvious one is to try to see if some math can be used to manipulate gender.\n\nTo find it I had VoiceDesign create 14 different voices for men and women each. When you then take the average woman minus the average man you can then move any voice along that direction:\n\n```\nwomen = torch.stack([fingerprint(f\"design/f{i}.wav\") for i in range(14)]).float()\nmen = torch.stack([fingerprint(f\"design/m{i}.wav\") for i in range(14)]).float()\nmiddle = (women.mean(0) + men.mean(0)) / 2\naxis = women.mean(0) - men.mean(0)\naxis = axis / axis.norm()\n\ndef move(xvec, t):\n \"\"\"Put the voice at position t along the gender axis, keep the rest.\"\"\"\n x = xvec.float()\n return (x + (t - (x - middle) @ axis) * axis).to(torch.bfloat16)\n```\n\nHere is the result for my clone:\n\nLoading the voice samples…\n\nAnyways, that’s that. If you want to play around with Qwen3 TTS I hope this was of some help, and maybe you got some ideas of what to do with the voice embeddings. Use it for good, not evil.\n\nThis entry was tagged [ai](https://lucumr.pocoo.org/tags/ai/) and [audio](https://lucumr.pocoo.org/tags/audio/)\n\n[copy as](https://lucumr.pocoo.org/2026/10/29/qwen3-custom-voices.md) / [view](https://lucumr.pocoo.org/2026/10/29/qwen3-custom-voices.md) markdown\n", "dateCreated": "2026-10-29T20:30:27.834895129+00:00", "date_created": "2026-10-29T00:00:00+00:00", "date_modified": "2026-10-29T00:00:00+00:00", "date_published": "2026-10-29T00:00:00+00:00", "description": "With my kids I’m currently building a little dungeron crawler and we’re trying to learn a bit about game development along the way. However given that we live in the modern times of increasing capabilities we wanted to get a narrator and…", "external_url": "https://lucumr.pocoo.org/2026/10/29/qwen3-custom-voices/", "git": { "commit": "20068632b65d2830a2d54f25ca1d90d697bea2c0", "path": "items/lucumr-pocoo-org/2026/10/2026-10-29-custom-voices-with-qwen3-tts.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/20068632b65d2830a2d54f25ca1d90d697bea2c0/items/lucumr-pocoo-org/2026/10/2026-10-29-custom-voices-with-qwen3-tts.md" }, "id": "urn:aggr:item:d80f51bdc7a11d99a94359259d9f9492b786b0bd", "isBasedOn": { "@id": "https://lucumr.pocoo.org/2026/10/29/qwen3-custom-voices/" }, "mainEntity": { "@id": "https://lucumr.pocoo.org/2026/10/29/qwen3-custom-voices/", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/lucumr-pocoo-org/2026-10-29-custom-voices-with-qwen3-tts/#webpage" }, "author": [ { "@type": "Person", "name": "Armin Ronacher" } ], "dateModified": "2026-10-29T00:00:00+00:00", "datePublished": "2026-10-29T00:00:00+00:00", "headline": "Custom Voices With Qwen3-TTS", "image": { "@type": "ImageObject", "caption": null, "height": 90, "url": "https://lucumr.pocoo.org/social/2026-10-29-qwen3-custom-voices-social.png", "width": 160 }, "timeRequired": "PT7M", "url": "https://lucumr.pocoo.org/2026/10/29/qwen3-custom-voices/", "wordCount": 1559 }, "name": "Archived snapshot: Custom Voices With Qwen3-TTS", "preview": { "alt": null, "color": null, "height": 90, "placeholder": { "data_url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII=", "hash": "" }, "url": "https://lucumr.pocoo.org/social/2026-10-29-qwen3-custom-voices-social.png", "width": 160 }, "reading_minutes": 7, "replicated_at": null, "source": "lucumr-pocoo-org", "source_name": "Armin Ronacher's Thoughts and Writings", "summary": null, "tags": [ "ai", "audio" ], "temporalCoverage": "2026-10-29T20:30:27.834895129+00:00", "title": "Custom Voices With Qwen3-TTS", "url": "https://aggr.aymericbeaumet.com/items/lucumr-pocoo-org/2026-10-29-custom-voices-with-qwen3-tts/", "word_count": 1559 }