{ "@context": "https://schema.org", "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2025-11-17-easily-build-and-share-rocm-kernels-with-hugging-face/#webpage", "@type": [ "WebPage", "ArchiveComponent" ], "articleBody": "[![Easily Build and Share ROCm Kernels with Hugging Face](https://huggingface.co/blog/assets/build-rocm-kernels/thumbnail.png)](https://huggingface.co/blog/assets/build-rocm-kernels/thumbnail.png)\n\n## Intoduction\n\nCustom kernels are the backbone of high-performance deep learning, enabling GPU operations tailored precisely to your workload; whether that’s image processing, tensor transformations, or other compute-heavy tasks. But compiling these kernels for the right architectures, wiring all the build flags, and integrating them cleanly into PyTorch extensions can quickly become a mess of CMake/Nix, compiler errors, and ABI issues, which is not fun. Hugging Face’s [**kernels**](https://github.com/huggingface/kernels) library makes it easy to build (with [**kernel-builder**](https://github.com/huggingface/kernels/tree/main/kernel-builder)) and share these kernels with the [**kernels-community**](https://huggingface.co/kernels-community), with support for multiple GPU and accelerator backends, including CUDA, ROCm, Metal, and XPU. This ensures your kernels are fast, portable, and seamlessly integrated with PyTorch.\n\nIn this guide, we focus exclusively on ROCm-compatible kernels and show how to build, test, and share them using [kernels](https://github.com/huggingface/kernels/tree/main). You’ll learn how to create kernels that run efficiently on AMD GPUs, along with best practices for reproducibility, packaging, and deployment.\n\nThis ROCm-specific walkthrough is a streamlined version of the original kernel-builder guide. If you’re looking for the broader CUDA-focused version, you can find it here: [A Guide to Building and Scaling Production-Ready CUDA Kernels](https://huggingface.co/blog/kernel-builder).\n\n## Build Steps\n\nWe will use the GEMM kernel from [RadeonFlow\\_Kernels](https://github.com/RadeonFlow/RadeonFlow_Kernels) as an example. If you want to go straight to the guide, [click here](https://huggingface.co/blog/build-rocm-kernels#step-1-project-structure).\n\n### About the kernel\n\n> This section was written by the **RadeonFlow GEMM** kernel authors to introduce the kernel.\\\n> Authors: [ColorsWind](https://huggingface.co/ColorsWind), [Zesen Liu](https://huggingface.co/ftyghome), and [Andy](https://huggingface.co/jpy794)\n\nThe **RadeonFlow GEMM** kernel is a high-performance, FP8 block-wise matrix multiplication implementation optimized for the AMD Instinct MI300X GPU. GEMM (General Matrix Multiplication) is the core building block behind most deep learning workloads: given two matrices A and B, you compute their product C = A × B. Here it’s implemented in FP8, a low-precision floating-point format that trades a bit of accuracy for much higher throughput and lower memory bandwidth. This kernel was developed for the [AMD Developer Challenge 2025](https://www.datamonsters.com/amd-developer-challenge-2025), it was awarded the 🏆 **Grand Prize** in **June 2025**, recognizing its excellence in performance and innovation on AMD hardware.\n\nThe kernel operates on quantized inputs using the `e4m3fnuz` floating-point format and applies per-block scaling to preserve accuracy during low-precision computation. The `e4m3fnuz` format is an FP8 variant with 4 exponent bits and 3 mantissa bits, designed to be efficient for neural network workloads. Because FP8 has a much smaller dynamic range than FP16/FP32, we apply per-block scaling factors (a\\_scale and b\\_scale) so that each block of values is rescaled into a numerically “comfortable” range before and after computation, which helps preserve accuracy despite the low precision. It takes the following arguments:\n\n```\n(a, b, a_scale, b_scale, c)\n```\n\nwhere `a` and `b` are the input matrices, `a_scale` and `b_scale` are the scaling factors for `a` and `b` respectively, and `c` is the output matrix:\n\n- `a` is K × M in e4m3fnuz\n- `b` is K × N in e4m3fnuz\n- `a_scale` is (K // 128) × M in fp32\n- `b_scale` is (K // 128) × (N // 128) in fp32\n- `c` is M × N in bf16\n\nThe kernel is precompiled for specific matrix shapes and assumes a transposed memory layout (as required by the competition). To support additional shapes or alternative memory layouts, you must modify the kernel launcher.\n\nSo now that we have a high-performance ROCm kernel, the natural question is: how do we integrate it into a real PyTorch workflow and share it with others? That’s exactly what we’ll cover next, using `kernels` to structure, build, and publish the ROCm kernel.\n\n> This is a fairly technical guide, but you can still follow it step by step without understanding every detail and everything will work fine. If you’re curious, you can always come back later to dig deeper into the concepts.\n\n### Step 1: Project Structure\n\nThe Hugging Face Kernel Builder expects your files to be organized like this:\n\n```\ngemm/\n├── build.toml\n├── gemm\n│ └── gemm_kernel.h\n├── flake.nix\n└── torch-ext\n ├── torch_binding.cpp\n ├── torch_binding.h\n └── gemm\n └── __init__.py\n```\n\n- **build.toml**: The project manifest; it’s the brain of the build process.\n- **gemm/**: Your raw CUDA source code where the GPU magic happens.\n- **flake.nix**: The key to a perfectly reproducible build environment.\n- **torch-ext/gemm/**: The Python wrapper for the raw PyTorch operators\n\nSometimes your project might depend on other files, like tests or helper scripts, and you can add them without any issues. In our case, our project will be structured like this:\n\n```\ngemm/\n├── build.toml\n├── gemm\n│ ├── gemm_kernel.h\n│ ├── gemm_kernel_legacy.h\n│ ├── transpose_kernel.h\n│ └── gemm_launcher.hip\n├── include\n│ ├── clangd_workaround.h\n│ ├── gpu_libs.h\n│ ├── gpu_types.h\n│ └── timer.h\n├── src/utils\n│ ├── arithmetic.h\n│ └── timer.hip\n├── tests/checker\n│ ├── checker.cpp\n│ ├── metrics.h\n│ └── checker.h\n├── flake.nix\n└── torch-ext\n ├── torch_binding.cpp\n ├── torch_binding.h\n └── gemm\n └── __init__.py\n```\n\nIf you look at the original files of the gemm kernel in the RadeonFlow Kernels, they are HIP source files with `.cpp` extensions. As a first step, you need to change these extensions to either .h or .hip depending on their content and usage:\n\n- Use `.h` for header files containing kernel declarations, inline functions, or template code that will be included in other files\n- Use `.hip` for implementation files containing HIP/GPU code that needs to be compiled separately (e.g., kernel launchers, device functions with complex implementations)\n\nIn our example, `gemm_kernel.h`, `gemm_kernel_legacy.h`, and `transpose_kernel.h` are header files, while `gemm_launcher.hip` is a HIP implementation file. This naming convention helps the kernel-builder ([`kernels/builder`](https://github.com/huggingface/kernels/tree/main/kernel-builder)) correctly identify and compile each file type.\n\n### Step 2: Configuration Files Setup\n\n#### The `build.toml` Manifest\n\nThis file orchestrates the entire build. It tells the kernel-builder what to compile and how everything connects.\n\n```toml\n[general]\nname = \"gemm\"\nuniversal = false\n\n[torch]\nsrc = [\n \"torch-ext/torch_binding.cpp\",\n \"torch-ext/torch_binding.h\",\n]\n\n[kernel.gemm]\nbackend = \"rocm\"\nrocm-archs = [\n \"gfx942\",\n]\n\ndepends = [\"torch\"]\n\nsrc = [\n \"include/clangd_workaround.h\",\n \"include/gpu_libs.h\",\n \"include/gpu_types.h\",\n \"include/timer.h\",\n \"gemm/gemm_kernel.h\",\n \"gemm/gemm_kernel_legacy.h\",\n \"gemm/gemm_launcher.hip\",\n \"gemm/transpose_kernel.h\",\n \"src/utils/arithmetic.h\",\n \"src/utils/timer.hip\",\n \"tests/checker/metrics.h\",\n]\n\ninclude = [\"include\"]\n```\n\n**general**\n\nThis section contains general project configuration settings.\n\n- **name** (required): The name of your project. This should match your kernel name and will be used for the Python package.\n- **universal** (optional): the kernel is a universal kernel when set to `true`. A universal kernel is a pure Python package (no compiled files). Universal kernels do not use the other sections described below. A good example of a universal kernel is a Triton kernel. Default: `false`\n\n**torch**\n\nThis section describes the Torch extension configuration. It defines the Python bindings that will expose your kernel to PyTorch.\n\n- **src** (required): A list of source files and headers for the PyTorch extension. In our case, this includes the C++ binding files that create the Python interface.\n\n**kernel.gemm**\n\nSpecification of a kernel named \"gemm\". You can define multiple kernel sections in the same build.toml file if you have multiple kernels.\n\n- **backend** (required): The compute backend for the kernel. We use \"rocm\" for AMD GPU support.\n- **rocm-archs** (required for ROCm): A list of ROCm architectures that the kernel should be compiled for. \"gfx942\" targets the MI300 series GPUs.\n- **depends** (required): A list of dependencies. We depend on \"torch\" to use PyTorch's tensor operations.\n- **include** (optional): Include directories relative to the project root. This helps the compiler find header files.\n\n#### The `flake.nix` Reproducibility File\n\nTo ensure anyone can build your kernel on any machine, we use a flake.nix file. It locks the exact version of the kernel-builder and its dependencies. (You can just copy and paste this example and change the description)\n\n```nix\n{\n description = \"Flake for GEMM kernel\";\n\n inputs = {\n kernel-builder.url = \"github:huggingface/kernels\";\n };\n\n outputs =\n {\n self,\n kernel-builder,\n }:\n\n kernel-builder.lib.genFlakeOutputs {\n inherit self;\n path = ./.;\n };\n}\n```\n\n#### Writing the Kernel\n\nNow for the GPU code. Inside `gemm/gemm_launcher.hip`, we define how the GEMM kernel is launched. Depending on the configuration, we either call the new optimized `gemm/gemm_kernel` or fall back to the legacy implementation (`gemm/gemm_kernel_legacy`).\n\n```c\n// ... previous includes and definitions\nextern \"C\" void run(\n void *a, void *b, void *as, void *bs, void *c,\n int m, int n, int k,\n PerfMetrics *metrics, hipStream_t job_stream0\n) {\n const __FP8_TYPE *a_ptr = static_cast(a);\n const __FP8_TYPE *b_ptr = static_cast(b);\n __BF16_TYPE *c_ptr = static_cast<__BF16_TYPE *>(c);\n const float *as_ptr = static_cast(as);\n const float *bs_ptr = static_cast(bs);\n\n KernelTimerScoped timer(timers, 2LL * m * n * k,\n metrics ? &metrics->entries[0].time : nullptr,\n metrics ? &metrics->entries[0].gflops : nullptr, job_stream0);\n\n // Dispatch GEMM to the fastest available implementation\n switch (pack_shape(m, n, k)) {\n DISPATCH_GEMM(1024, 1536, 7168, 256, 128, 128, 4, 2, 512, 4, 16);\n DISPATCH_GEMM(6144, 7168, 2304, 256, 128, 128, 4, 2, 512, 1, 16);\n default: {\n printf(\"Error: Unsupported shape M=%d, K=%d, N=%d\\n\", m, k, n);\n abort();\n }\n }\n}\n// ...\n```\n\n#### Registering a Native PyTorch Operator\n\nThis step is key. We’re not just making the function available in Python; we’re turning it into a native PyTorch operator. That means it becomes a first-class part of PyTorch itself, accessible through `torch.ops`.\n\nThe file `torch-ext/torch_binding.cpp` handles this registration.\n\n```c\n#include \n#include \n#include \n\n#include \"registration.h\"\n#include \"torch_binding.h\"\n\n// Forward declaration of the C function from gemm_launcher.hip\nextern \"C\" {\n struct PerfMetrics;\n void run(void *a, void *b, void *as, void *bs, void *c, int m, int n, int k, PerfMetrics *metrics, hipStream_t job_stream0);\n}\n\nvoid gemm(torch::Tensor &out, torch::Tensor const &a, torch::Tensor const &b, \n torch::Tensor const &as, torch::Tensor const &bs) {\n \n // Validate tensor properties\n TORCH_CHECK(a.device().is_cuda(), \"Input tensor a must be on GPU device\");\n TORCH_CHECK(b.device().is_cuda(), \"Input tensor b must be on GPU device\");\n TORCH_CHECK(as.device().is_cuda(), \"Scale tensor as must be on GPU device\");\n TORCH_CHECK(bs.device().is_cuda(), \"Scale tensor bs must be on GPU device\");\n TORCH_CHECK(out.device().is_cuda(), \"Output tensor out must be on GPU device\");\n \n TORCH_CHECK(a.is_contiguous(), \"Input tensor a must be contiguous\");\n TORCH_CHECK(b.is_contiguous(), \"Input tensor b must be contiguous\");\n TORCH_CHECK(as.is_contiguous(), \"Scale tensor as must be contiguous\");\n TORCH_CHECK(bs.is_contiguous(), \"Scale tensor bs must be contiguous\");\n TORCH_CHECK(out.is_contiguous(), \"Output tensor out must be contiguous\");\n \n // Get matrix dimensions from tensor shapes\n // Assuming a is [M, K], b is [K, N], out is [M, N]\n int M = a.size(0);\n int K = a.size(1);\n int N = b.size(1);\n \n TORCH_CHECK(b.size(0) == K, \"Matrix dimensions mismatch: a.size(1) != b.size(0)\");\n TORCH_CHECK(out.size(0) == M, \"Output tensor dimension mismatch: out.size(0) != M\");\n TORCH_CHECK(out.size(1) == N, \"Output tensor dimension mismatch: out.size(1) != N\");\n \n // Use default HIP stream (stream 0)\n const hipStream_t stream = 0;\n \n // Call the C function\n run(a.data_ptr(), b.data_ptr(), as.data_ptr(), bs.data_ptr(), out.data_ptr(),\n M, N, K, nullptr, stream);\n}\n\nTORCH_LIBRARY_EXPAND(TORCH_EXTENSION_NAME, ops) {\n ops.def(\"gemm(Tensor! out, Tensor a, Tensor b, Tensor a_scale, Tensor b_scale) -> ()\");\n ops.impl(\"gemm\", torch::kCUDA, &gemm);\n}\n\nREGISTER_EXTENSION(TORCH_EXTENSION_NAME)\n```\n\nThe `torch_binding.h` file contains function declarations. For instance, the `gemm` kernel has the following declaration in `torch_binding.h`:\n\n```h\n#pragma once\n\n#include \n\nvoid gemm(torch::Tensor &out, torch::Tensor const &a, torch::Tensor const &b, \n torch::Tensor const &as, torch::Tensor const &bs);\n```\n\n#### Setting up the `__init__.py` wrapper\n\nIn `torch-ext/gemm/` we need an `__init__.py` file to make this directory a Python package and to expose our custom operator in a user-friendly way.\n\n```python\nfrom typing import Optional\nimport torch\nfrom ._ops import ops\n\ndef gemm(a: torch.Tensor, b: torch.Tensor, as_: torch.Tensor, bs: torch.Tensor, \n out: Optional[torch.Tensor] = None) -> torch.Tensor:\n \n if out is None:\n # Create output tensor with appropriate shape and dtype\n M, K = a.shape\n K_b, N = b.shape\n assert K == K_b, f\"Matrix dimension mismatch: A has {K} cols, B has {K_b} rows\"\n \n # Output should be BF16 type on the same device as inputs\n out = torch.empty((M, N), dtype=torch.bfloat16, device=a.device)\n \n ops.gemm(out, a, b, as_, bs)\n return out\n```\n\n### Step 3: Building the Kernel\n\nThe kernel builder uses Nix for building kernels. You can build or run the kernels directly if you have Nix installed on your system. We recommend installing Nix in the following way:\n\n- **Linux**: use the [official Nix installer](https://nixos.org/download/).\n- **macOS**: use the [Determinate Nix installer](https://docs.determinate.systems/determinate-nix/). In addition, Xcode 16.x is currently required to build kernels.\n\n#### Getting Started with Nix\n\nFirst of all, run this:\n\n```bash\nnix flake update\n```\n\nThis generates a `flake.lock` file that pins the kernel builder and all its transitive dependencies. Commit both `flake.nix` and `flake.lock` to your repository to ensure that kernel builds are reproducible.\n\nSince the kernel builder depends on many packages (e.g., every supported PyTorch version), it is recommended to enable the Hugging Face cache to avoid expensive rebuilds:\n\n```bash\n# Install cachix and configure the cache\ncachix use huggingface\n```\n\nOr run it once without installing cachix permanently:\n\n```bash\n# Use cachix without installing it\nnix run nixpkgs#cachix -- use huggingface\n```\n\n#### Building Kernels with Nix\n\nA kernel that has a `flake.nix` file can be built with the build-and-copy command:\n\n```bash\ncd Build_RadeonFlow_Kernels/gemm\nnix build . -L\n```\n\nThe compiled kernel will then be in the local `build/` directory.\n\n#### Development Shell for Local Development\n\nThe kernel-builder provides shells for developing kernels. In such a shell, all required dependencies are available, as well as `build2cmake` for generating project files:\n\n```bash\n$ nix develop\n$ build2cmake generate-torch build.toml\n$ cmake -B build-ext\n$ cmake --build build-ext\n```\n\nIf you want to test the kernel as a Python package, you can do so. `nix develop` will automatically create a virtual environment in `.venv` and activate it:\n\n```bash\n$ nix develop\n$ build2cmake generate-torch build.toml\n$ pip install --no-build-isolation -e .\n```\n\nDevelopment shells are available for every build configuration. For instance, you can get a Torch 2.7 development shell with ROCm 6.3 using:\n\n```bash\n$ rm -rf .venv # Remove existing venv if any\n$ nix develop .#devShells.torch27-cxx11-rocm63-x86_64-linux\n```\n\n### Step 4: Uploading the kernel to the Hub\n\nNow that we built our kernel, we can test it and upload it to the Hub.\n\n#### Building the Kernel for All PyTorch and ROCm Versions\n\nOne small thing we'll want to do before we share is clean up all of the development artifacts that were generated during the build process to avoid uploading unnecessary files.\n\n```bash\nbuild2cmake clean build.toml \n```\n\nTo build the kernel for all supported versions of PyTorch and ROCm, the kernel-builder tool automates the process:\n\n```bash\n# Outside of the dev shell, run the following command\n# if you are inside of the sandbox you can leave with `exit`\nnix build . -L\n```\n\n> **Note:**\\\n> This process may take a while, as it will build the kernel for all supported versions of PyTorch and ROCm.\\\n> The output will be in the `result` directory.\n\nThe last step is to move the results into the expected build directory (this is where the kernels library will look for them).\n\n```bash\nmkdir -p build\nrsync -av --delete --chmod=Du+w,Fu+w result/ build/\n```\n\n#### Pushing to the Hugging Face Hub\n\nPushing the build artifacts to the Hub will make it straightforward for other developers to use your kernel. We can use the `kernels upload` command for this:\n\n```bash\nkernels upload --repo_id hub-username/img2gray\n```\n\nYou can also follow a standard git-based process for the upload.\n\nFirst, create a new repo:\n\n```bash\nhf repo create gemm\n```\n\n> Make sure you are logged in to the Hugging Face Hub using huggingface-cli login.\n\nNow, in your project directory, connect your project to the new repository and push your code:\n\n```bash\n# Initialize git and connect to the Hugging Face Hub\ngit init\ngit remote add origin https://huggingface.co//gemm\n\n# Pull the changes (just the default .gitattributes file)\ngit pull origin main\ngit xet install\ngit checkout -b main\n\n# Update to use Xet for the binary files\ngit xet track \"*.so\"\n\n# Add and commit your changes (being careful to only include the necessary files\n# since our build2cmake command generated a lot of dev-specific files)\ngit add \\\n build/ gemm/ include/ src/utils tests/checker \\\n torch-ext/torch_binding.cpp torch-ext/torch_binding.h torch-ext/gemm \\\n flake.nix flake.lock build.toml\n\ngit commit -m \"feat: Created a compliant gemm kernel\"\ngit push -u origin main\n```\n\nFantastic! Your kernel is now on the Hugging Face Hub, ready for others to use and fully compliant with the kernels library.\n\n### Step 5: Let's use it :)\n\nWith the **kernels** library, you don't \"install\" the kernel in the traditional sense. You load it directly from its Hub repository, which automatically registers the new operator.\n\n```python\nimport torch\nfrom kernels import get_kernel\n\n# Load the kernel from the Hub\ngemm = get_kernel(\"kernels-community/gemm\")\n\n# Matrix dimensions (must be supported - see gemm_launcher.cpp)\nM, N, K = 1024, 1536, 7168\nQUANT_SIZE = 128\n\n# Setup device\ndevice = torch.device(\"cuda\")\n\n# Create inputs - kernel expects A:(K,M), B:(K,N)\nA_fp32 = torch.randn(M, K, device=device)\nB_fp32 = torch.randn(K, N, device=device)\n\n# Convert to FP8\nA_fp8 = A_fp32.to(torch.float8_e4m3fnuz)\nB_fp8 = B_fp32.to(torch.float8_e4m3fnuz)\n\n# Create scale factors (uniform scaling)\nA_scale = torch.ones(K // QUANT_SIZE, M, device=device, dtype=torch.float32)\nB_scale = torch.ones(K // QUANT_SIZE, N // QUANT_SIZE, device=device, dtype=torch.float32)\n\nC = torch.zeros(M, N, device=device, dtype=torch.bfloat16)\n\n# Use the kernel\nresult = gemm.gemm(A_fp8, B_fp8, A_scale, B_scale, C)\n```\n\nThat's it! Your ROCm kernel is now ready to use from the Hugging Face Hub.\n\n## Conclusion\n\nBuilding and sharing ROCm kernels with the Hugging Face is now easier than ever. With a clean, reproducible workflow powered by Nix and seamless integration into PyTorch, developers can focus on optimizing performance rather than setup. Once built, your custom kernel can be shared on the Hugging Face Hub; making it instantly accessible to the community and usable across projects with just a few lines of code. 🚀\n\n## Related Libraries & Hub\n\n- [kernels](https://github.com/huggingface/kernels) – Library to build, manage and load kernels from the Hub. It contains the [kernel-builder](https://github.com/huggingface/kernels/tree/main/kernel-builder) tooling.\n- [Kernels Community Hub](https://huggingface.co/kernels-community) – Share and discover kernels from the community.\n", "authors": [], "captured_at": "2026-10-04T12:50:30.804479567+00:00", "category": "ai", "content_html": "

\"Easily

\n

Intoduction

\n

Custom kernels are the backbone of high-performance deep learning, enabling GPU operations tailored precisely to your workload; whether that’s image processing, tensor transformations, or other compute-heavy tasks. But compiling these kernels for the right architectures, wiring all the build flags, and integrating them cleanly into PyTorch extensions can quickly become a mess of CMake/Nix, compiler errors, and ABI issues, which is not fun. Hugging Face’s kernels library makes it easy to build (with kernel-builder) and share these kernels with the kernels-community, with support for multiple GPU and accelerator backends, including CUDA, ROCm, Metal, and XPU. This ensures your kernels are fast, portable, and seamlessly integrated with PyTorch.

\n

In this guide, we focus exclusively on ROCm-compatible kernels and show how to build, test, and share them using kernels. You’ll learn how to create kernels that run efficiently on AMD GPUs, along with best practices for reproducibility, packaging, and deployment.

\n

This ROCm-specific walkthrough is a streamlined version of the original kernel-builder guide. If you’re looking for the broader CUDA-focused version, you can find it here: A Guide to Building and Scaling Production-Ready CUDA Kernels.

\n

Build Steps

\n

We will use the GEMM kernel from RadeonFlow_Kernels as an example. If you want to go straight to the guide, click here.

\n

About the kernel

\n
\n

This section was written by the RadeonFlow GEMM kernel authors to introduce the kernel.
\nAuthors: ColorsWind, Zesen Liu, and Andy

\n
\n

The RadeonFlow GEMM kernel is a high-performance, FP8 block-wise matrix multiplication implementation optimized for the AMD Instinct MI300X GPU. GEMM (General Matrix Multiplication) is the core building block behind most deep learning workloads: given two matrices A and B, you compute their product C = A × B. Here it’s implemented in FP8, a low-precision floating-point format that trades a bit of accuracy for much higher throughput and lower memory bandwidth. This kernel was developed for the AMD Developer Challenge 2025, it was awarded the 🏆 Grand Prize in June 2025, recognizing its excellence in performance and innovation on AMD hardware.

\n

The kernel operates on quantized inputs using the e4m3fnuz floating-point format and applies per-block scaling to preserve accuracy during low-precision computation. The e4m3fnuz format is an FP8 variant with 4 exponent bits and 3 mantissa bits, designed to be efficient for neural network workloads. Because FP8 has a much smaller dynamic range than FP16/FP32, we apply per-block scaling factors (a_scale and b_scale) so that each block of values is rescaled into a numerically “comfortable” range before and after computation, which helps preserve accuracy despite the low precision. It takes the following arguments:

\n
(a, b, a_scale, b_scale, c)\n
\n

where a and b are the input matrices, a_scale and b_scale are the scaling factors for a and b respectively, and c is the output matrix:

\n
    \n
  • a is K × M in e4m3fnuz
    • \n
  • b is K × N in e4m3fnuz
    • \n
  • a_scale is (K // 128) × M in fp32
    • \n
  • b_scale is (K // 128) × (N // 128) in fp32
    • \n
  • c is M × N in bf16
    • \n
\n

The kernel is precompiled for specific matrix shapes and assumes a transposed memory layout (as required by the competition). To support additional shapes or alternative memory layouts, you must modify the kernel launcher.

\n

So now that we have a high-performance ROCm kernel, the natural question is: how do we integrate it into a real PyTorch workflow and share it with others? That’s exactly what we’ll cover next, using kernels to structure, build, and publish the ROCm kernel.

\n
\n

This is a fairly technical guide, but you can still follow it step by step without understanding every detail and everything will work fine. If you’re curious, you can always come back later to dig deeper into the concepts.

\n
\n

Step 1: Project Structure

\n

The Hugging Face Kernel Builder expects your files to be organized like this:

\n
gemm/\n├── build.toml\n├── gemm\n│   └── gemm_kernel.h\n├── flake.nix\n└── torch-ext\n    ├── torch_binding.cpp\n    ├── torch_binding.h\n    └── gemm\n        └── __init__.py\n
\n
    \n
  • build.toml: The project manifest; it’s the brain of the build process.
    • \n
  • gemm/: Your raw CUDA source code where the GPU magic happens.
    • \n
  • flake.nix: The key to a perfectly reproducible build environment.
    • \n
  • torch-ext/gemm/: The Python wrapper for the raw PyTorch operators
    • \n
\n

Sometimes your project might depend on other files, like tests or helper scripts, and you can add them without any issues. In our case, our project will be structured like this:

\n
gemm/\n├── build.toml\n├── gemm\n│   ├── gemm_kernel.h\n│   ├── gemm_kernel_legacy.h\n│   ├── transpose_kernel.h\n│   └── gemm_launcher.hip\n├── include\n│   ├── clangd_workaround.h\n│   ├── gpu_libs.h\n│   ├── gpu_types.h\n│   └── timer.h\n├── src/utils\n│   ├── arithmetic.h\n│   └── timer.hip\n├── tests/checker\n│   ├── checker.cpp\n│   ├── metrics.h\n│   └── checker.h\n├── flake.nix\n└── torch-ext\n    ├── torch_binding.cpp\n    ├── torch_binding.h\n    └── gemm\n        └── __init__.py\n
\n

If you look at the original files of the gemm kernel in the RadeonFlow Kernels, they are HIP source files with .cpp extensions. As a first step, you need to change these extensions to either .h or .hip depending on their content and usage:

\n
    \n
  • Use .h for header files containing kernel declarations, inline functions, or template code that will be included in other files
    • \n
  • Use .hip for implementation files containing HIP/GPU code that needs to be compiled separately (e.g., kernel launchers, device functions with complex implementations)
    • \n
\n

In our example, gemm_kernel.h, gemm_kernel_legacy.h, and transpose_kernel.h are header files, while gemm_launcher.hip is a HIP implementation file. This naming convention helps the kernel-builder (kernels/builder) correctly identify and compile each file type.

\n

Step 2: Configuration Files Setup

\n

The build.toml Manifest

\n

This file orchestrates the entire build. It tells the kernel-builder what to compile and how everything connects.

\n
[general]\nname = "gemm"\nuniversal = false\n\n[torch]\nsrc = [\n  "torch-ext/torch_binding.cpp",\n  "torch-ext/torch_binding.h",\n]\n\n[kernel.gemm]\nbackend = "rocm"\nrocm-archs = [\n    "gfx942",\n]\n\ndepends = ["torch"]\n\nsrc = [\n  "include/clangd_workaround.h",\n  "include/gpu_libs.h",\n  "include/gpu_types.h",\n  "include/timer.h",\n  "gemm/gemm_kernel.h",\n  "gemm/gemm_kernel_legacy.h",\n  "gemm/gemm_launcher.hip",\n  "gemm/transpose_kernel.h",\n  "src/utils/arithmetic.h",\n  "src/utils/timer.hip",\n  "tests/checker/metrics.h",\n]\n\ninclude = ["include"]\n
\n

general

\n

This section contains general project configuration settings.

\n
    \n
  • name (required): The name of your project. This should match your kernel name and will be used for the Python package.
    • \n
  • universal (optional): the kernel is a universal kernel when set to true. A universal kernel is a pure Python package (no compiled files). Universal kernels do not use the other sections described below. A good example of a universal kernel is a Triton kernel. Default: false
    • \n
\n

torch

\n

This section describes the Torch extension configuration. It defines the Python bindings that will expose your kernel to PyTorch.

\n
    \n
  • src (required): A list of source files and headers for the PyTorch extension. In our case, this includes the C++ binding files that create the Python interface.
    • \n
\n

kernel.gemm

\n

Specification of a kernel named "gemm". You can define multiple kernel sections in the same build.toml file if you have multiple kernels.

\n
    \n
  • backend (required): The compute backend for the kernel. We use "rocm" for AMD GPU support.
    • \n
  • rocm-archs (required for ROCm): A list of ROCm architectures that the kernel should be compiled for. "gfx942" targets the MI300 series GPUs.
    • \n
  • depends (required): A list of dependencies. We depend on "torch" to use PyTorch's tensor operations.
    • \n
  • include (optional): Include directories relative to the project root. This helps the compiler find header files.
    • \n
\n

The flake.nix Reproducibility File

\n

To ensure anyone can build your kernel on any machine, we use a flake.nix file. It locks the exact version of the kernel-builder and its dependencies. (You can just copy and paste this example and change the description)

\n
{\n  description = "Flake for GEMM kernel";\n\n  inputs = {\n    kernel-builder.url = "github:huggingface/kernels";\n  };\n\n  outputs =\n    {\n      self,\n      kernel-builder,\n    }:\n\n    kernel-builder.lib.genFlakeOutputs {\n      inherit self;\n      path = ./.;\n    };\n}\n
\n

Writing the Kernel

\n

Now for the GPU code. Inside gemm/gemm_launcher.hip, we define how the GEMM kernel is launched. Depending on the configuration, we either call the new optimized gemm/gemm_kernel or fall back to the legacy implementation (gemm/gemm_kernel_legacy).

\n
// ... previous includes and definitions\nextern "C" void run(\n    void *a, void *b, void *as, void *bs, void *c,\n    int m, int n, int k,\n    PerfMetrics *metrics, hipStream_t job_stream0\n) {\n    const __FP8_TYPE *a_ptr = static_cast<const __FP8_TYPE *>(a);\n    const __FP8_TYPE *b_ptr = static_cast<const __FP8_TYPE *>(b);\n    __BF16_TYPE *c_ptr = static_cast<__BF16_TYPE *>(c);\n    const float *as_ptr = static_cast<const float *>(as);\n    const float *bs_ptr = static_cast<const float *>(bs);\n\n    KernelTimerScoped timer(timers, 2LL * m * n * k,\n        metrics ? &metrics->entries[0].time : nullptr,\n        metrics ? &metrics->entries[0].gflops : nullptr, job_stream0);\n\n    // Dispatch GEMM to the fastest available implementation\n    switch (pack_shape(m, n, k)) {\n        DISPATCH_GEMM(1024, 1536, 7168, 256, 128, 128, 4, 2, 512, 4, 16);\n        DISPATCH_GEMM(6144, 7168, 2304, 256, 128, 128, 4, 2, 512, 1, 16);\n        default: {\n            printf("Error: Unsupported shape M=%d, K=%d, N=%d\\n", m, k, n);\n            abort();\n        }\n    }\n}\n// ...\n
\n

Registering a Native PyTorch Operator

\n

This step is key. We’re not just making the function available in Python; we’re turning it into a native PyTorch operator. That means it becomes a first-class part of PyTorch itself, accessible through torch.ops.

\n

The file torch-ext/torch_binding.cpp handles this registration.

\n
#include <torch/all.h>\n#include <torch/library.h>\n#include <hip/hip_runtime.h>\n\n#include "registration.h"\n#include "torch_binding.h"\n\n// Forward declaration of the C function from gemm_launcher.hip\nextern "C" {\n    struct PerfMetrics;\n    void run(void *a, void *b, void *as, void *bs, void *c, int m, int n, int k, PerfMetrics *metrics, hipStream_t job_stream0);\n}\n\nvoid gemm(torch::Tensor &out, torch::Tensor const &a, torch::Tensor const &b, \n          torch::Tensor const &as, torch::Tensor const &bs) {\n    \n    // Validate tensor properties\n    TORCH_CHECK(a.device().is_cuda(), "Input tensor a must be on GPU device");\n    TORCH_CHECK(b.device().is_cuda(), "Input tensor b must be on GPU device");\n    TORCH_CHECK(as.device().is_cuda(), "Scale tensor as must be on GPU device");\n    TORCH_CHECK(bs.device().is_cuda(), "Scale tensor bs must be on GPU device");\n    TORCH_CHECK(out.device().is_cuda(), "Output tensor out must be on GPU device");\n    \n    TORCH_CHECK(a.is_contiguous(), "Input tensor a must be contiguous");\n    TORCH_CHECK(b.is_contiguous(), "Input tensor b must be contiguous");\n    TORCH_CHECK(as.is_contiguous(), "Scale tensor as must be contiguous");\n    TORCH_CHECK(bs.is_contiguous(), "Scale tensor bs must be contiguous");\n    TORCH_CHECK(out.is_contiguous(), "Output tensor out must be contiguous");\n    \n    // Get matrix dimensions from tensor shapes\n    // Assuming a is [M, K], b is [K, N], out is [M, N]\n    int M = a.size(0);\n    int K = a.size(1);\n    int N = b.size(1);\n    \n    TORCH_CHECK(b.size(0) == K, "Matrix dimensions mismatch: a.size(1) != b.size(0)");\n    TORCH_CHECK(out.size(0) == M, "Output tensor dimension mismatch: out.size(0) != M");\n    TORCH_CHECK(out.size(1) == N, "Output tensor dimension mismatch: out.size(1) != N");\n    \n    // Use default HIP stream (stream 0)\n    const hipStream_t stream = 0;\n    \n    // Call the C function\n    run(a.data_ptr(), b.data_ptr(), as.data_ptr(), bs.data_ptr(), out.data_ptr(),\n        M, N, K, nullptr, stream);\n}\n\nTORCH_LIBRARY_EXPAND(TORCH_EXTENSION_NAME, ops) {\n  ops.def("gemm(Tensor! out, Tensor a, Tensor b, Tensor a_scale, Tensor b_scale) -> ()");\n  ops.impl("gemm", torch::kCUDA, &gemm);\n}\n\nREGISTER_EXTENSION(TORCH_EXTENSION_NAME)\n
\n

The torch_binding.h file contains function declarations. For instance, the gemm kernel has the following declaration in torch_binding.h:

\n
#pragma once\n\n#include <torch/torch.h>\n\nvoid gemm(torch::Tensor &out, torch::Tensor const &a, torch::Tensor const &b, \n          torch::Tensor const &as, torch::Tensor const &bs);\n
\n

Setting up the __init__.py wrapper

\n

In torch-ext/gemm/ we need an __init__.py file to make this directory a Python package and to expose our custom operator in a user-friendly way.

\n
from typing import Optional\nimport torch\nfrom ._ops import ops\n\ndef gemm(a: torch.Tensor, b: torch.Tensor, as_: torch.Tensor, bs: torch.Tensor, \n         out: Optional[torch.Tensor] = None) -> torch.Tensor:\n         \n    if out is None:\n        # Create output tensor with appropriate shape and dtype\n        M, K = a.shape\n        K_b, N = b.shape\n        assert K == K_b, f"Matrix dimension mismatch: A has {K} cols, B has {K_b} rows"\n        \n        # Output should be BF16 type on the same device as inputs\n        out = torch.empty((M, N), dtype=torch.bfloat16, device=a.device)\n    \n    ops.gemm(out, a, b, as_, bs)\n    return out\n
\n

Step 3: Building the Kernel

\n

The kernel builder uses Nix for building kernels. You can build or run the kernels directly if you have Nix installed on your system. We recommend installing Nix in the following way:

\n\n

Getting Started with Nix

\n

First of all, run this:

\n
nix flake update\n
\n

This generates a flake.lock file that pins the kernel builder and all its transitive dependencies. Commit both flake.nix and flake.lock to your repository to ensure that kernel builds are reproducible.

\n

Since the kernel builder depends on many packages (e.g., every supported PyTorch version), it is recommended to enable the Hugging Face cache to avoid expensive rebuilds:

\n
# Install cachix and configure the cache\ncachix use huggingface\n
\n

Or run it once without installing cachix permanently:

\n
# Use cachix without installing it\nnix run nixpkgs#cachix -- use huggingface\n
\n

Building Kernels with Nix

\n

A kernel that has a flake.nix file can be built with the build-and-copy command:

\n
cd Build_RadeonFlow_Kernels/gemm\nnix build . -L\n
\n

The compiled kernel will then be in the local build/ directory.

\n

Development Shell for Local Development

\n

The kernel-builder provides shells for developing kernels. In such a shell, all required dependencies are available, as well as build2cmake for generating project files:

\n
$ nix develop\n$ build2cmake generate-torch build.toml\n$ cmake -B build-ext\n$ cmake --build build-ext\n
\n

If you want to test the kernel as a Python package, you can do so. nix develop will automatically create a virtual environment in .venv and activate it:

\n
$ nix develop\n$ build2cmake generate-torch build.toml\n$ pip install --no-build-isolation -e .\n
\n

Development shells are available for every build configuration. For instance, you can get a Torch 2.7 development shell with ROCm 6.3 using:

\n
$ rm -rf .venv  # Remove existing venv if any\n$ nix develop .#devShells.torch27-cxx11-rocm63-x86_64-linux\n
\n

Step 4: Uploading the kernel to the Hub

\n

Now that we built our kernel, we can test it and upload it to the Hub.

\n

Building the Kernel for All PyTorch and ROCm Versions

\n

One small thing we'll want to do before we share is clean up all of the development artifacts that were generated during the build process to avoid uploading unnecessary files.

\n
build2cmake clean build.toml \n
\n

To build the kernel for all supported versions of PyTorch and ROCm, the kernel-builder tool automates the process:

\n
# Outside of the dev shell, run the following command\n# if you are inside of the sandbox you can leave with `exit`\nnix build . -L\n
\n
\n

Note:
\nThis process may take a while, as it will build the kernel for all supported versions of PyTorch and ROCm.
\nThe output will be in the result directory.

\n
\n

The last step is to move the results into the expected build directory (this is where the kernels library will look for them).

\n
mkdir -p build\nrsync -av --delete --chmod=Du+w,Fu+w result/ build/\n
\n

Pushing to the Hugging Face Hub

\n

Pushing the build artifacts to the Hub will make it straightforward for other developers to use your kernel. We can use the kernels upload command for this:

\n
kernels upload <path_to_kernel> --repo_id hub-username/img2gray\n
\n

You can also follow a standard git-based process for the upload.

\n

First, create a new repo:

\n
hf repo create gemm\n
\n
\n

Make sure you are logged in to the Hugging Face Hub using huggingface-cli login.

\n
\n

Now, in your project directory, connect your project to the new repository and push your code:

\n
# Initialize git and connect to the Hugging Face Hub\ngit init\ngit remote add origin https://huggingface.co/<your-username>/gemm\n\n# Pull the changes (just the default .gitattributes file)\ngit pull origin main\ngit xet install\ngit checkout -b main\n\n# Update to use Xet for the binary files\ngit xet track "*.so"\n\n# Add and commit your changes (being careful to only include the necessary files\n# since our build2cmake command generated a lot of dev-specific files)\ngit add \\\n  build/ gemm/ include/ src/utils tests/checker \\\n  torch-ext/torch_binding.cpp torch-ext/torch_binding.h torch-ext/gemm \\\n  flake.nix flake.lock build.toml\n\ngit commit -m "feat: Created a compliant gemm kernel"\ngit push -u origin main\n
\n

Fantastic! Your kernel is now on the Hugging Face Hub, ready for others to use and fully compliant with the kernels library.

\n

Step 5: Let's use it :)

\n

With the kernels library, you don't "install" the kernel in the traditional sense. You load it directly from its Hub repository, which automatically registers the new operator.

\n
import torch\nfrom kernels import get_kernel\n\n# Load the kernel from the Hub\ngemm = get_kernel("kernels-community/gemm")\n\n# Matrix dimensions (must be supported - see gemm_launcher.cpp)\nM, N, K = 1024, 1536, 7168\nQUANT_SIZE = 128\n\n# Setup device\ndevice = torch.device("cuda")\n\n# Create inputs - kernel expects A:(K,M), B:(K,N)\nA_fp32 = torch.randn(M, K, device=device)\nB_fp32 = torch.randn(K, N, device=device)\n\n# Convert to FP8\nA_fp8 = A_fp32.to(torch.float8_e4m3fnuz)\nB_fp8 = B_fp32.to(torch.float8_e4m3fnuz)\n\n# Create scale factors (uniform scaling)\nA_scale = torch.ones(K // QUANT_SIZE, M, device=device, dtype=torch.float32)\nB_scale = torch.ones(K // QUANT_SIZE, N // QUANT_SIZE, device=device, dtype=torch.float32)\n\nC = torch.zeros(M, N, device=device, dtype=torch.bfloat16)\n\n# Use the kernel\nresult = gemm.gemm(A_fp8, B_fp8, A_scale, B_scale, C)\n
\n

That's it! Your ROCm kernel is now ready to use from the Hugging Face Hub.

\n

Conclusion

\n

Building and sharing ROCm kernels with the Hugging Face is now easier than ever. With a clean, reproducible workflow powered by Nix and seamless integration into PyTorch, developers can focus on optimizing performance rather than setup. Once built, your custom kernel can be shared on the Hugging Face Hub; making it instantly accessible to the community and usable across projects with just a few lines of code. 🚀

\n

Related Libraries & Hub

\n\n", "content_markdown": "[![Easily Build and Share ROCm Kernels with Hugging Face](https://huggingface.co/blog/assets/build-rocm-kernels/thumbnail.png)](https://huggingface.co/blog/assets/build-rocm-kernels/thumbnail.png)\n\n## Intoduction\n\nCustom kernels are the backbone of high-performance deep learning, enabling GPU operations tailored precisely to your workload; whether that’s image processing, tensor transformations, or other compute-heavy tasks. But compiling these kernels for the right architectures, wiring all the build flags, and integrating them cleanly into PyTorch extensions can quickly become a mess of CMake/Nix, compiler errors, and ABI issues, which is not fun. Hugging Face’s [**kernels**](https://github.com/huggingface/kernels) library makes it easy to build (with [**kernel-builder**](https://github.com/huggingface/kernels/tree/main/kernel-builder)) and share these kernels with the [**kernels-community**](https://huggingface.co/kernels-community), with support for multiple GPU and accelerator backends, including CUDA, ROCm, Metal, and XPU. This ensures your kernels are fast, portable, and seamlessly integrated with PyTorch.\n\nIn this guide, we focus exclusively on ROCm-compatible kernels and show how to build, test, and share them using [kernels](https://github.com/huggingface/kernels/tree/main). You’ll learn how to create kernels that run efficiently on AMD GPUs, along with best practices for reproducibility, packaging, and deployment.\n\nThis ROCm-specific walkthrough is a streamlined version of the original kernel-builder guide. If you’re looking for the broader CUDA-focused version, you can find it here: [A Guide to Building and Scaling Production-Ready CUDA Kernels](https://huggingface.co/blog/kernel-builder).\n\n## Build Steps\n\nWe will use the GEMM kernel from [RadeonFlow\\_Kernels](https://github.com/RadeonFlow/RadeonFlow_Kernels) as an example. If you want to go straight to the guide, [click here](https://huggingface.co/blog/build-rocm-kernels#step-1-project-structure).\n\n### About the kernel\n\n> This section was written by the **RadeonFlow GEMM** kernel authors to introduce the kernel.\\\n> Authors: [ColorsWind](https://huggingface.co/ColorsWind), [Zesen Liu](https://huggingface.co/ftyghome), and [Andy](https://huggingface.co/jpy794)\n\nThe **RadeonFlow GEMM** kernel is a high-performance, FP8 block-wise matrix multiplication implementation optimized for the AMD Instinct MI300X GPU. GEMM (General Matrix Multiplication) is the core building block behind most deep learning workloads: given two matrices A and B, you compute their product C = A × B. Here it’s implemented in FP8, a low-precision floating-point format that trades a bit of accuracy for much higher throughput and lower memory bandwidth. This kernel was developed for the [AMD Developer Challenge 2025](https://www.datamonsters.com/amd-developer-challenge-2025), it was awarded the 🏆 **Grand Prize** in **June 2025**, recognizing its excellence in performance and innovation on AMD hardware.\n\nThe kernel operates on quantized inputs using the `e4m3fnuz` floating-point format and applies per-block scaling to preserve accuracy during low-precision computation. The `e4m3fnuz` format is an FP8 variant with 4 exponent bits and 3 mantissa bits, designed to be efficient for neural network workloads. Because FP8 has a much smaller dynamic range than FP16/FP32, we apply per-block scaling factors (a\\_scale and b\\_scale) so that each block of values is rescaled into a numerically “comfortable” range before and after computation, which helps preserve accuracy despite the low precision. It takes the following arguments:\n\n```\n(a, b, a_scale, b_scale, c)\n```\n\nwhere `a` and `b` are the input matrices, `a_scale` and `b_scale` are the scaling factors for `a` and `b` respectively, and `c` is the output matrix:\n\n- `a` is K × M in e4m3fnuz\n- `b` is K × N in e4m3fnuz\n- `a_scale` is (K // 128) × M in fp32\n- `b_scale` is (K // 128) × (N // 128) in fp32\n- `c` is M × N in bf16\n\nThe kernel is precompiled for specific matrix shapes and assumes a transposed memory layout (as required by the competition). To support additional shapes or alternative memory layouts, you must modify the kernel launcher.\n\nSo now that we have a high-performance ROCm kernel, the natural question is: how do we integrate it into a real PyTorch workflow and share it with others? That’s exactly what we’ll cover next, using `kernels` to structure, build, and publish the ROCm kernel.\n\n> This is a fairly technical guide, but you can still follow it step by step without understanding every detail and everything will work fine. If you’re curious, you can always come back later to dig deeper into the concepts.\n\n### Step 1: Project Structure\n\nThe Hugging Face Kernel Builder expects your files to be organized like this:\n\n```\ngemm/\n├── build.toml\n├── gemm\n│ └── gemm_kernel.h\n├── flake.nix\n└── torch-ext\n ├── torch_binding.cpp\n ├── torch_binding.h\n └── gemm\n └── __init__.py\n```\n\n- **build.toml**: The project manifest; it’s the brain of the build process.\n- **gemm/**: Your raw CUDA source code where the GPU magic happens.\n- **flake.nix**: The key to a perfectly reproducible build environment.\n- **torch-ext/gemm/**: The Python wrapper for the raw PyTorch operators\n\nSometimes your project might depend on other files, like tests or helper scripts, and you can add them without any issues. In our case, our project will be structured like this:\n\n```\ngemm/\n├── build.toml\n├── gemm\n│ ├── gemm_kernel.h\n│ ├── gemm_kernel_legacy.h\n│ ├── transpose_kernel.h\n│ └── gemm_launcher.hip\n├── include\n│ ├── clangd_workaround.h\n│ ├── gpu_libs.h\n│ ├── gpu_types.h\n│ └── timer.h\n├── src/utils\n│ ├── arithmetic.h\n│ └── timer.hip\n├── tests/checker\n│ ├── checker.cpp\n│ ├── metrics.h\n│ └── checker.h\n├── flake.nix\n└── torch-ext\n ├── torch_binding.cpp\n ├── torch_binding.h\n └── gemm\n └── __init__.py\n```\n\nIf you look at the original files of the gemm kernel in the RadeonFlow Kernels, they are HIP source files with `.cpp` extensions. As a first step, you need to change these extensions to either .h or .hip depending on their content and usage:\n\n- Use `.h` for header files containing kernel declarations, inline functions, or template code that will be included in other files\n- Use `.hip` for implementation files containing HIP/GPU code that needs to be compiled separately (e.g., kernel launchers, device functions with complex implementations)\n\nIn our example, `gemm_kernel.h`, `gemm_kernel_legacy.h`, and `transpose_kernel.h` are header files, while `gemm_launcher.hip` is a HIP implementation file. This naming convention helps the kernel-builder ([`kernels/builder`](https://github.com/huggingface/kernels/tree/main/kernel-builder)) correctly identify and compile each file type.\n\n### Step 2: Configuration Files Setup\n\n#### The `build.toml` Manifest\n\nThis file orchestrates the entire build. It tells the kernel-builder what to compile and how everything connects.\n\n```toml\n[general]\nname = \"gemm\"\nuniversal = false\n\n[torch]\nsrc = [\n \"torch-ext/torch_binding.cpp\",\n \"torch-ext/torch_binding.h\",\n]\n\n[kernel.gemm]\nbackend = \"rocm\"\nrocm-archs = [\n \"gfx942\",\n]\n\ndepends = [\"torch\"]\n\nsrc = [\n \"include/clangd_workaround.h\",\n \"include/gpu_libs.h\",\n \"include/gpu_types.h\",\n \"include/timer.h\",\n \"gemm/gemm_kernel.h\",\n \"gemm/gemm_kernel_legacy.h\",\n \"gemm/gemm_launcher.hip\",\n \"gemm/transpose_kernel.h\",\n \"src/utils/arithmetic.h\",\n \"src/utils/timer.hip\",\n \"tests/checker/metrics.h\",\n]\n\ninclude = [\"include\"]\n```\n\n**general**\n\nThis section contains general project configuration settings.\n\n- **name** (required): The name of your project. This should match your kernel name and will be used for the Python package.\n- **universal** (optional): the kernel is a universal kernel when set to `true`. A universal kernel is a pure Python package (no compiled files). Universal kernels do not use the other sections described below. A good example of a universal kernel is a Triton kernel. Default: `false`\n\n**torch**\n\nThis section describes the Torch extension configuration. It defines the Python bindings that will expose your kernel to PyTorch.\n\n- **src** (required): A list of source files and headers for the PyTorch extension. In our case, this includes the C++ binding files that create the Python interface.\n\n**kernel.gemm**\n\nSpecification of a kernel named \"gemm\". You can define multiple kernel sections in the same build.toml file if you have multiple kernels.\n\n- **backend** (required): The compute backend for the kernel. We use \"rocm\" for AMD GPU support.\n- **rocm-archs** (required for ROCm): A list of ROCm architectures that the kernel should be compiled for. \"gfx942\" targets the MI300 series GPUs.\n- **depends** (required): A list of dependencies. We depend on \"torch\" to use PyTorch's tensor operations.\n- **include** (optional): Include directories relative to the project root. This helps the compiler find header files.\n\n#### The `flake.nix` Reproducibility File\n\nTo ensure anyone can build your kernel on any machine, we use a flake.nix file. It locks the exact version of the kernel-builder and its dependencies. (You can just copy and paste this example and change the description)\n\n```nix\n{\n description = \"Flake for GEMM kernel\";\n\n inputs = {\n kernel-builder.url = \"github:huggingface/kernels\";\n };\n\n outputs =\n {\n self,\n kernel-builder,\n }:\n\n kernel-builder.lib.genFlakeOutputs {\n inherit self;\n path = ./.;\n };\n}\n```\n\n#### Writing the Kernel\n\nNow for the GPU code. Inside `gemm/gemm_launcher.hip`, we define how the GEMM kernel is launched. Depending on the configuration, we either call the new optimized `gemm/gemm_kernel` or fall back to the legacy implementation (`gemm/gemm_kernel_legacy`).\n\n```c\n// ... previous includes and definitions\nextern \"C\" void run(\n void *a, void *b, void *as, void *bs, void *c,\n int m, int n, int k,\n PerfMetrics *metrics, hipStream_t job_stream0\n) {\n const __FP8_TYPE *a_ptr = static_cast(a);\n const __FP8_TYPE *b_ptr = static_cast(b);\n __BF16_TYPE *c_ptr = static_cast<__BF16_TYPE *>(c);\n const float *as_ptr = static_cast(as);\n const float *bs_ptr = static_cast(bs);\n\n KernelTimerScoped timer(timers, 2LL * m * n * k,\n metrics ? &metrics->entries[0].time : nullptr,\n metrics ? &metrics->entries[0].gflops : nullptr, job_stream0);\n\n // Dispatch GEMM to the fastest available implementation\n switch (pack_shape(m, n, k)) {\n DISPATCH_GEMM(1024, 1536, 7168, 256, 128, 128, 4, 2, 512, 4, 16);\n DISPATCH_GEMM(6144, 7168, 2304, 256, 128, 128, 4, 2, 512, 1, 16);\n default: {\n printf(\"Error: Unsupported shape M=%d, K=%d, N=%d\\n\", m, k, n);\n abort();\n }\n }\n}\n// ...\n```\n\n#### Registering a Native PyTorch Operator\n\nThis step is key. We’re not just making the function available in Python; we’re turning it into a native PyTorch operator. That means it becomes a first-class part of PyTorch itself, accessible through `torch.ops`.\n\nThe file `torch-ext/torch_binding.cpp` handles this registration.\n\n```c\n#include \n#include \n#include \n\n#include \"registration.h\"\n#include \"torch_binding.h\"\n\n// Forward declaration of the C function from gemm_launcher.hip\nextern \"C\" {\n struct PerfMetrics;\n void run(void *a, void *b, void *as, void *bs, void *c, int m, int n, int k, PerfMetrics *metrics, hipStream_t job_stream0);\n}\n\nvoid gemm(torch::Tensor &out, torch::Tensor const &a, torch::Tensor const &b, \n torch::Tensor const &as, torch::Tensor const &bs) {\n \n // Validate tensor properties\n TORCH_CHECK(a.device().is_cuda(), \"Input tensor a must be on GPU device\");\n TORCH_CHECK(b.device().is_cuda(), \"Input tensor b must be on GPU device\");\n TORCH_CHECK(as.device().is_cuda(), \"Scale tensor as must be on GPU device\");\n TORCH_CHECK(bs.device().is_cuda(), \"Scale tensor bs must be on GPU device\");\n TORCH_CHECK(out.device().is_cuda(), \"Output tensor out must be on GPU device\");\n \n TORCH_CHECK(a.is_contiguous(), \"Input tensor a must be contiguous\");\n TORCH_CHECK(b.is_contiguous(), \"Input tensor b must be contiguous\");\n TORCH_CHECK(as.is_contiguous(), \"Scale tensor as must be contiguous\");\n TORCH_CHECK(bs.is_contiguous(), \"Scale tensor bs must be contiguous\");\n TORCH_CHECK(out.is_contiguous(), \"Output tensor out must be contiguous\");\n \n // Get matrix dimensions from tensor shapes\n // Assuming a is [M, K], b is [K, N], out is [M, N]\n int M = a.size(0);\n int K = a.size(1);\n int N = b.size(1);\n \n TORCH_CHECK(b.size(0) == K, \"Matrix dimensions mismatch: a.size(1) != b.size(0)\");\n TORCH_CHECK(out.size(0) == M, \"Output tensor dimension mismatch: out.size(0) != M\");\n TORCH_CHECK(out.size(1) == N, \"Output tensor dimension mismatch: out.size(1) != N\");\n \n // Use default HIP stream (stream 0)\n const hipStream_t stream = 0;\n \n // Call the C function\n run(a.data_ptr(), b.data_ptr(), as.data_ptr(), bs.data_ptr(), out.data_ptr(),\n M, N, K, nullptr, stream);\n}\n\nTORCH_LIBRARY_EXPAND(TORCH_EXTENSION_NAME, ops) {\n ops.def(\"gemm(Tensor! out, Tensor a, Tensor b, Tensor a_scale, Tensor b_scale) -> ()\");\n ops.impl(\"gemm\", torch::kCUDA, &gemm);\n}\n\nREGISTER_EXTENSION(TORCH_EXTENSION_NAME)\n```\n\nThe `torch_binding.h` file contains function declarations. For instance, the `gemm` kernel has the following declaration in `torch_binding.h`:\n\n```h\n#pragma once\n\n#include \n\nvoid gemm(torch::Tensor &out, torch::Tensor const &a, torch::Tensor const &b, \n torch::Tensor const &as, torch::Tensor const &bs);\n```\n\n#### Setting up the `__init__.py` wrapper\n\nIn `torch-ext/gemm/` we need an `__init__.py` file to make this directory a Python package and to expose our custom operator in a user-friendly way.\n\n```python\nfrom typing import Optional\nimport torch\nfrom ._ops import ops\n\ndef gemm(a: torch.Tensor, b: torch.Tensor, as_: torch.Tensor, bs: torch.Tensor, \n out: Optional[torch.Tensor] = None) -> torch.Tensor:\n \n if out is None:\n # Create output tensor with appropriate shape and dtype\n M, K = a.shape\n K_b, N = b.shape\n assert K == K_b, f\"Matrix dimension mismatch: A has {K} cols, B has {K_b} rows\"\n \n # Output should be BF16 type on the same device as inputs\n out = torch.empty((M, N), dtype=torch.bfloat16, device=a.device)\n \n ops.gemm(out, a, b, as_, bs)\n return out\n```\n\n### Step 3: Building the Kernel\n\nThe kernel builder uses Nix for building kernels. You can build or run the kernels directly if you have Nix installed on your system. We recommend installing Nix in the following way:\n\n- **Linux**: use the [official Nix installer](https://nixos.org/download/).\n- **macOS**: use the [Determinate Nix installer](https://docs.determinate.systems/determinate-nix/). In addition, Xcode 16.x is currently required to build kernels.\n\n#### Getting Started with Nix\n\nFirst of all, run this:\n\n```bash\nnix flake update\n```\n\nThis generates a `flake.lock` file that pins the kernel builder and all its transitive dependencies. Commit both `flake.nix` and `flake.lock` to your repository to ensure that kernel builds are reproducible.\n\nSince the kernel builder depends on many packages (e.g., every supported PyTorch version), it is recommended to enable the Hugging Face cache to avoid expensive rebuilds:\n\n```bash\n# Install cachix and configure the cache\ncachix use huggingface\n```\n\nOr run it once without installing cachix permanently:\n\n```bash\n# Use cachix without installing it\nnix run nixpkgs#cachix -- use huggingface\n```\n\n#### Building Kernels with Nix\n\nA kernel that has a `flake.nix` file can be built with the build-and-copy command:\n\n```bash\ncd Build_RadeonFlow_Kernels/gemm\nnix build . -L\n```\n\nThe compiled kernel will then be in the local `build/` directory.\n\n#### Development Shell for Local Development\n\nThe kernel-builder provides shells for developing kernels. In such a shell, all required dependencies are available, as well as `build2cmake` for generating project files:\n\n```bash\n$ nix develop\n$ build2cmake generate-torch build.toml\n$ cmake -B build-ext\n$ cmake --build build-ext\n```\n\nIf you want to test the kernel as a Python package, you can do so. `nix develop` will automatically create a virtual environment in `.venv` and activate it:\n\n```bash\n$ nix develop\n$ build2cmake generate-torch build.toml\n$ pip install --no-build-isolation -e .\n```\n\nDevelopment shells are available for every build configuration. For instance, you can get a Torch 2.7 development shell with ROCm 6.3 using:\n\n```bash\n$ rm -rf .venv # Remove existing venv if any\n$ nix develop .#devShells.torch27-cxx11-rocm63-x86_64-linux\n```\n\n### Step 4: Uploading the kernel to the Hub\n\nNow that we built our kernel, we can test it and upload it to the Hub.\n\n#### Building the Kernel for All PyTorch and ROCm Versions\n\nOne small thing we'll want to do before we share is clean up all of the development artifacts that were generated during the build process to avoid uploading unnecessary files.\n\n```bash\nbuild2cmake clean build.toml \n```\n\nTo build the kernel for all supported versions of PyTorch and ROCm, the kernel-builder tool automates the process:\n\n```bash\n# Outside of the dev shell, run the following command\n# if you are inside of the sandbox you can leave with `exit`\nnix build . -L\n```\n\n> **Note:**\\\n> This process may take a while, as it will build the kernel for all supported versions of PyTorch and ROCm.\\\n> The output will be in the `result` directory.\n\nThe last step is to move the results into the expected build directory (this is where the kernels library will look for them).\n\n```bash\nmkdir -p build\nrsync -av --delete --chmod=Du+w,Fu+w result/ build/\n```\n\n#### Pushing to the Hugging Face Hub\n\nPushing the build artifacts to the Hub will make it straightforward for other developers to use your kernel. We can use the `kernels upload` command for this:\n\n```bash\nkernels upload --repo_id hub-username/img2gray\n```\n\nYou can also follow a standard git-based process for the upload.\n\nFirst, create a new repo:\n\n```bash\nhf repo create gemm\n```\n\n> Make sure you are logged in to the Hugging Face Hub using huggingface-cli login.\n\nNow, in your project directory, connect your project to the new repository and push your code:\n\n```bash\n# Initialize git and connect to the Hugging Face Hub\ngit init\ngit remote add origin https://huggingface.co//gemm\n\n# Pull the changes (just the default .gitattributes file)\ngit pull origin main\ngit xet install\ngit checkout -b main\n\n# Update to use Xet for the binary files\ngit xet track \"*.so\"\n\n# Add and commit your changes (being careful to only include the necessary files\n# since our build2cmake command generated a lot of dev-specific files)\ngit add \\\n build/ gemm/ include/ src/utils tests/checker \\\n torch-ext/torch_binding.cpp torch-ext/torch_binding.h torch-ext/gemm \\\n flake.nix flake.lock build.toml\n\ngit commit -m \"feat: Created a compliant gemm kernel\"\ngit push -u origin main\n```\n\nFantastic! Your kernel is now on the Hugging Face Hub, ready for others to use and fully compliant with the kernels library.\n\n### Step 5: Let's use it :)\n\nWith the **kernels** library, you don't \"install\" the kernel in the traditional sense. You load it directly from its Hub repository, which automatically registers the new operator.\n\n```python\nimport torch\nfrom kernels import get_kernel\n\n# Load the kernel from the Hub\ngemm = get_kernel(\"kernels-community/gemm\")\n\n# Matrix dimensions (must be supported - see gemm_launcher.cpp)\nM, N, K = 1024, 1536, 7168\nQUANT_SIZE = 128\n\n# Setup device\ndevice = torch.device(\"cuda\")\n\n# Create inputs - kernel expects A:(K,M), B:(K,N)\nA_fp32 = torch.randn(M, K, device=device)\nB_fp32 = torch.randn(K, N, device=device)\n\n# Convert to FP8\nA_fp8 = A_fp32.to(torch.float8_e4m3fnuz)\nB_fp8 = B_fp32.to(torch.float8_e4m3fnuz)\n\n# Create scale factors (uniform scaling)\nA_scale = torch.ones(K // QUANT_SIZE, M, device=device, dtype=torch.float32)\nB_scale = torch.ones(K // QUANT_SIZE, N // QUANT_SIZE, device=device, dtype=torch.float32)\n\nC = torch.zeros(M, N, device=device, dtype=torch.bfloat16)\n\n# Use the kernel\nresult = gemm.gemm(A_fp8, B_fp8, A_scale, B_scale, C)\n```\n\nThat's it! Your ROCm kernel is now ready to use from the Hugging Face Hub.\n\n## Conclusion\n\nBuilding and sharing ROCm kernels with the Hugging Face is now easier than ever. With a clean, reproducible workflow powered by Nix and seamless integration into PyTorch, developers can focus on optimizing performance rather than setup. Once built, your custom kernel can be shared on the Hugging Face Hub; making it instantly accessible to the community and usable across projects with just a few lines of code. 🚀\n\n## Related Libraries & Hub\n\n- [kernels](https://github.com/huggingface/kernels) – Library to build, manage and load kernels from the Hub. It contains the [kernel-builder](https://github.com/huggingface/kernels/tree/main/kernel-builder) tooling.\n- [Kernels Community Hub](https://huggingface.co/kernels-community) – Share and discover kernels from the community.\n", "dateCreated": "2026-10-04T12:50:30.804479567+00:00", "date_created": "2025-11-17T00:00:00+00:00", "date_modified": "2025-11-17T00:00:00+00:00", "date_published": "2025-11-17T00:00:00+00:00", "description": "Intoduction Custom kernels are the backbone of high-performance deep learning, enabling GPU operations tailored precisely to your workload; whether that’s image processing, tensor transformations, or other compute-heavy tasks. But…", "external_url": "https://huggingface.co/blog/build-rocm-kernels", "git": { "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e", "path": "items/huggingface-co/2025/11/2025-11-17-easily-build-and-share-rocm-kernels-with-hugging-face.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/huggingface-co/2025/11/2025-11-17-easily-build-and-share-rocm-kernels-with-hugging-face.md" }, "id": "urn:aggr:item:1ce15d1454391202f5e5f6fae07fd1be1162d35b", "isBasedOn": { "@id": "https://huggingface.co/blog/build-rocm-kernels" }, "mainEntity": { "@id": "https://huggingface.co/blog/build-rocm-kernels", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2025-11-17-easily-build-and-share-rocm-kernels-with-hugging-face/#webpage" }, "dateModified": "2025-11-17T00:00:00+00:00", "datePublished": "2025-11-17T00:00:00+00:00", "headline": "Easily Build and Share ROCm Kernels with Hugging Face", "timeRequired": "PT13M", "url": "https://huggingface.co/blog/build-rocm-kernels", "wordCount": 2851 }, "name": "Archived snapshot: Easily Build and Share ROCm Kernels with Hugging Face", "preview": null, "reading_minutes": 13, "replicated_at": null, "source": "huggingface-co", "source_name": "Hugging Face - Blog", "summary": null, "tags": [], "temporalCoverage": "2026-10-04T12:50:30.804479567+00:00", "title": "Easily Build and Share ROCm Kernels with Hugging Face", "url": "https://aggr.aymericbeaumet.com/items/huggingface-co/2025-11-17-easily-build-and-share-rocm-kernels-with-hugging-face/", "word_count": 2851 }