![\"\"](\"https://substackcdn.com/image/fetch/$s_!_APX!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbb3d5266-b65a-45ea-9f8f-98d7d8038b8e_3432x1536.png\")
Previously, I compared the most notable open-weight architectures of 2025 in The Big LLM Architecture Comparison. Then, I zoomed in and discussed the various architecture components in From GPT-2 to gpt-oss: Analyzing the Architectural Advances on a conceptual level.
\nSince all good things come in threes, before covering some of the noteworthy research highlights of this summer, I wanted to now dive into these architectures hands-on, in code. By following along, you will understand how it actually works under the hood and gain building blocks you can adapt for your own experiments or projects.
\nFor this, I picked Qwen3 ( initially released in May and updated in July) because it is one of the most widely liked and used open-weight model families as of this writing.
\nThe reasons why Qwen3 models are so popular are, in my view, as follows:
\nA developer- and commercially friendly open-source ( Apache License v2.0) without any strings attached beyond the original open-source license terms (some other open-weight LLMs impose additional usage limits)
\nThe performance is really good; for example, as of this writing, the open-weight 235B-Instruct variant is ranked 8 on the LMArena leaderboard, tied with the proprietary Claude Opus 4. The only 2 other open-weight LLMs that rank higher are DeepSeek 3.1 (3x larger) and Kimi K2 (4x larger). On September 5th, Qwen3 released a 1T parameter “max” variant on their platform that beats Kimi K2, DeepSeek 3.1, and Claude Opus 4 on all major benchmarks; however, this model is closed-source for now.
\nThere are many different model sizes available for different compute budgets and use-cases, from 0.6B dense models to 480B parameter Mixture-of-Experts models.
\nThis is going to be a long article due to the from-scratch code in pure PyTorch. While the code sections may look verbose, I hope that they help explain the building blocks better than conceptual figures alone!
\nTip 1: If you are reading this article in your email inbox, the narrow line width may cause code snippets to wrap awkwardly. For a better experience, I recommend opening it in your web browser.
\nTip 2: You can use the table of contents on the left side of the website for easier navigation between sections.
\n
Figure 1: Preview of the Qwen3 Dense and Mixture-of-Experts architectures discussed and (re)implemented in pure PyTorch in this article.
\n", "content_markdown": "Previously, I compared the most notable open-weight architectures of 2025 in [The Big LLM Architecture Comparison](https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison). Then, I zoomed in and discussed the various architecture components in [From GPT-2 to gpt-oss: Analyzing the Architectural Advances](https://magazine.sebastianraschka.com/p/from-gpt-2-to-gpt-oss-analyzing-the) on a conceptual level.\n\nSince all good things come in threes, before covering some of the noteworthy research highlights of this summer, I wanted to now dive into these architectures hands-on, in code. By following along, you will understand how it actually works under the hood and gain building blocks you can adapt for your own experiments or projects.\n\nFor this, I picked Qwen3 ( [initially released in May](https://arxiv.org/abs/2505.09388) and updated in July) because it is one of the most widely liked and used open-weight model families as of this writing.\n\nThe reasons why Qwen3 models are so popular are, in my view, as follows:\n\n1. A developer- and commercially friendly open-source ( [Apache License v2.0](https://huggingface.co/Qwen/Qwen3-0.6B/blob/main/LICENSE)) without any strings attached beyond the original open-source license terms (some other open-weight LLMs impose additional usage limits)\n\n2. The performance is really good; for example, as of this writing, the open-weight 235B-Instruct variant is ranked 8 on the [LMArena leaderboard](https://lmarena.ai/leaderboard/text), tied with the proprietary Claude Opus 4. The only 2 other open-weight LLMs that rank higher are DeepSeek 3.1 (3x larger) and Kimi K2 (4x larger). On September 5th, [Qwen3 released a 1T parameter “max” variant](https://x.com/Alibaba_Qwen/status/1963991502440562976) on their platform that beats Kimi K2, DeepSeek 3.1, and Claude Opus 4 on all major benchmarks; however, this model is closed-source for now.\n\n3. There are many different model sizes available for different compute budgets and use-cases, from 0.6B dense models to 480B parameter Mixture-of-Experts models.\n\nThis is going to be a long article due to the from-scratch code in pure PyTorch. While the code sections may look verbose, I hope that they help explain the building blocks better than conceptual figures alone!\n\n**Tip 1:** If you are reading this article in your email inbox, the narrow line width may cause code snippets to wrap awkwardly. For a better experience, I recommend [opening it in your web browser](https://magazine.sebastianraschka.com/p/qwen3-from-scratch).\n\n**Tip 2:** You can use the table of contents on the left side of the website for easier navigation between sections.\n\n[](https://substackcdn.com/image/fetch/$s_!_APX!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbb3d5266-b65a-45ea-9f8f-98d7d8038b8e_3432x1536.png)\n\nFigure 1: Preview of the Qwen3 Dense and Mixture-of-Experts architectures discussed and (re)implemented in pure PyTorch in this article.\n", "dateCreated": "2026-09-27T19:29:15.293681927+00:00", "date_created": "2025-09-06T11:10:21+00:00", "date_modified": "2025-09-06T11:10:21+00:00", "date_published": "2025-09-06T11:10:21+00:00", "description": "A Detailed Look at One of the Leading Open-Source LLMs", "external_url": "https://magazine.sebastianraschka.com/p/qwen3-from-scratch", "git": { "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e", "path": "items/magazine-sebastianraschka-com/2025/09/2025-09-06-understanding-and-implementing-qwen3-from-scratch.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/magazine-sebastianraschka-com/2025/09/2025-09-06-understanding-and-implementing-qwen3-from-scratch.md" }, "id": "urn:aggr:item:309108d8fbeaa83eba6d367ad8ff4dfe1c24732b", "isBasedOn": { "@id": "https://magazine.sebastianraschka.com/p/qwen3-from-scratch" }, "mainEntity": { "@id": "https://magazine.sebastianraschka.com/p/qwen3-from-scratch", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/magazine-sebastianraschka-com/2025-09-06-understanding-and-implementing-qwen3-from-scratch/#webpage" }, "author": [ { "@type": "Person", "name": "Sebastian Raschka, PhD" } ], "dateModified": "2025-09-06T11:10:21+00:00", "datePublished": "2025-09-06T11:10:21+00:00", "headline": "Understanding and Implementing Qwen3 From Scratch", "timeRequired": "PT2M", "url": "https://magazine.sebastianraschka.com/p/qwen3-from-scratch", "wordCount": 401 }, "name": "Archived snapshot: Understanding and Implementing Qwen3 From Scratch", "preview": null, "reading_minutes": 2, "replicated_at": null, "source": "magazine-sebastianraschka-com", "source_name": "Ahead of AI", "summary": "A Detailed Look at One of the Leading Open-Source LLMs", "tags": [], "temporalCoverage": "2026-09-27T19:29:15.293681927+00:00", "title": "Understanding and Implementing Qwen3 From Scratch", "url": "https://aggr.aymericbeaumet.com/items/magazine-sebastianraschka-com/2025-09-06-understanding-and-implementing-qwen3-from-scratch/", "word_count": 401 }