{ "@context": "https://schema.org", "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-01-28-we-got-claude-to-build-cuda-kernels-and-teach-open-models/#webpage", "@type": [ "WebPage", "ArchiveComponent" ], "articleBody": "The best thing about agent skills is *upskilling* your agents on hard problems. There are two ways to look at that:\n\n1. You can take Opus 4.5 or other SOTA models and tackle the hardest problems out there.\n2. You can take models that run on your laptop and upskill them to harder problems. In this blog post, we’ll show you how to take on the latter.\n\nThis blog post walks through the process of using a new tool, `upskill`, to generate and evaluate agent skills with large models and use them with smaller models. We will benchmark `upskill` on the task of writing CUDA kernels for [`diffusers`](https://huggingface.co/docs/diffusers/en/index) models, but the process is generally useful for cutting costs, or using smaller models on hard and domain-specific problems.\n\n## What are agent skills?\n\nIn case you missed it, agent skills are taking the coding agent game by storm. In fact, they’re a straightforward concept to define model context as files, like instructions as markdown and code as scripts. The file format makes them easy to generate, share, and review. In short, they’re a practical medium to share capabilities across models and tools, and they're most useful in specific domains or hard problems. Not stuff the model can do well anyway.\n\nThis post showcases this process by using Claude to generate a Skill file that can be used by open source models for a complex and specialized task: write CUDA kernels. We first tried a simple skill based on existing documentation, and we found that it improved performance for some others, but not all. In fact, it could even degrade performance or increase token usage for some models. Check out the plot below to see the performance of the model with and without the basic skill.\n\n[![plot of model performance](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/upskill-blog/plot.png)](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/upskill-blog/plot.png)\n\nNow, let's walk through how you can use `upskill` to upskill your agents on hard problems, and measure performance.\n\n## 1. Get the teacher (Claude Opus 4.5) to build a kernel\n\nFirst, we use Claude Code to build a kernel interactively and export the trace. We worked through the process by instructing, validating, and adding documentation links. This somewhat naive process is important to reveal the models' initial challenges. In fact, you can iterate on this multiple times, by trying to solve the task with draft versions of the skill, and experimenting with smaller models. Each time, you can instruct the agent to improve the skill and test it on the smaller model.\n\nHere's an [example of the skill](https://huggingface.co/hf-skills/h100-diffusers-kernel-builder) that we created and have been using to build kernels. We started from this [agent trace](https://huggingface.co/hf-skills/h100-diffusers-kernel-builder/blob/main/agent-trace.txt) where the agent was able to build a kernel, but not without some help.\n\n## 2. Make an agent skill from the trace\n\nOnce the teacher model has performed the task, we need them to make a skill. There are a number of effective ways to do this.\n\n- Within the same session, instruct the agent to create a skill file for the task it just completed.\n- Use [Anthropic ‘skill creator’ skill](https://github.com/anthropics/skills/blob/main/skills/skill-creator/SKILL.md) either within the agent session or with an exported trace and a new agent session.\n- Use the `upskill` tool to create a skill based on the trace.\n\nIn most cases, the first 2 options result in functional skills. However, the performance of an agent with the skill is unknown. That’s where `upskill` is useful, because it will also generate test cases for your skill based on the trace. It then compares the results under both scenarios: using the trace, or applying the skill. We see below that the original model (Claude Opus) met the same performance with and without the skill. This means the skill captured the task *for this model*. Great!\n\n[![terminal evaluation](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/upskill-blog/terminal.png)](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/upskill-blog/terminal.png)\n\n## 3. Take your skill to an open source, smaller, or cheaper model\n\nFinally, we need to transfer our newly created skill to the tool or model we intend to use. Most tools like `codex`, `cursor`, and `opencode` have settled on a consistent format for skills, which is a directory at `{agent}/skills/{skill_name}/SKILL.md` , so we just need to copy the skill directory to this location.\n\nWith `upskill` we can pass a skill and a set of models to the `eval` command and `upskill` will run the test cases on those models with and without the skill to compare performance. We can see here that the skill increases accuracy on some open models, but not on all.\n\n[![performance evaluation](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/upskill-blog/accuracy.png)](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/upskill-blog/accuracy.png)\n\nIn this case, we might want to iterate further on the `gpt-oss` skills by regenerating the skill. We can do `upskill generate --from {skill}`.\n\nThere is more to agent skills than model performance. Often agents can reach a given accuracy with or without a skill, they just need to consume more tokens to get there. For recurring tasks, we want to optimize agents to use less tokens to achieve the same accuracy. The results below reveal another dimension to the skill. Some models are significantly reducing their performance token usage, whilst others are using more tokens **with** the skill. For example, with [`moonshotai/Kimi-K2-Thinking`](https://huggingface.co/moonshotai/Kimi-K2-Thinking) the skill is clearly effective in terms of accuracy and token usage. However, for Claude Opus 4.5 there is no clear performance increase and an increase in token usage, so you would not want to use this skill with Claude Opus 4.5.\n\n[![token usage](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/upskill-blog/tokens.png)](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/upskill-blog/tokens.png)\n\n**tldr;** try out and evaluate models with the skills you create. Use `upskill eval` or a similar tool to evaluate the models performance with and without skills.\n\nThat’s the high level end to end of upskilling your coding agents on hard problems. Try out upskill now like this:\n\n```shell\n# install upskill\npip install upskill\n\n# or use uvx \nuvx upskill --help\n\n# generate a skill based on an agent trace\nupskill generate \"write nvidia kernels\" --from ./trace.md\n\u000b# evaluate models on a skill\nupskill eval ./skills/my-skill/ --model haiku --model sonnet\n\n# generate skills for local models\nupskill generate \"parse YAML\" \n --model opus \n --eval-model \"unsloth/GLM-4.7-Flash-GGUF:Q4_0\" \n --eval-base-url http://localhost:8080/v1\n```\n\n## Deep dive tutorial into building kernels with agent skills\n\nWe have a high level understanding of how we can upskill an agent. Let’s now look at the use case we solved for writing CUDA kernels.\n\nWe didn’t just want to write kernel code, but understand the full kernel-builder workflow: project structure, `build.toml` configuration, architecture-specific optimizations, and PyTorch bindings. This tutorial shows how upskill creates validated skills that actually work.\n\nThe [`kernel-builder-cuda-kernels`](https://github.com/burtenshaw/kernel-skill) skill teaches Claude everything it needs to know about CUDA development: which GPU architecture to target, how to structure a kernel-builder project, when to use shared memory versus registers, and how to write PyTorch bindings.\n\nWith this skill, you can tell Claude things like:\n\n```\nBuild a fused LayerNorm + GELU kernel optimized for H100.\n```\n\nAnd Claude will create the complete project structure, CUDA implementation, and build configuration—following the exact conventions that kernel-builder expects.\n\nThis isn't about generating boilerplate. The skill encodes domain expertise: H100 uses compute capability 9.0, shared memory should be aligned to 128 bytes, async memory copies require `__CUDA_ARCH__ >= 900`. Knowledge that would take hours to gather from documentation gets packaged into ~500 tokens that load on demand.\n\n## Setup and Install\n\nInstall upskill:\n\n```shell\npip install upskill\n# or use uvx for one-off runs\nuvx upskill --help\n```\n\nSet your API key:\n\n```shell\nexport ANTHROPIC_API_KEY=sk-ant-...\nexport HF_TOKEN=hf_...\n```\n\nThat's it. Upskill uses Anthropic Claude Opus-4.5 model by default but also supports OpenAI and local models via OpenAI-compatible endpoints as generators. We want to use the more expensive and higher quality models to generate skills, and the smaller ones to use them. Think robin hood.\n\n## Skill Generation\n\nLet's walk through generating a skill that teaches agents how to build CUDA kernels with HuggingFace's [`kernels`](https://github.com/huggingface/kernels) library.\n\n### Generate the Skill\n\nStart with a clear task description:\n\n```shell\nupskill generate \"build optimized CUDA kernels for PyTorch using HuggingFace kernel-builder\"\n```\n\nAbove we used upskill, but it could in fact be any agent or chat tool and an exported trace.\n\n```shell\nupskill generate \"write kernels\" --from .md\n```\n\nAlso, we could start from an existing skill and add to it:\n\n```shell\nupskill generate \"add more error handling and edge cases\" \n --from ./skills/kernel-builder-cuda-kernels/\n```\n\nupskill loads the existing skill, applies your improvements, and re-evaluates to ensure the changes help.\n\nupskill creates a skill, generates test cases, evaluates performance, and refines based on failures:\n\n```\nGenerating skill with sonnet...\nGenerating test cases...\nEvaluating on sonnet... (attempt 1)\n 60% -> 95% (+35%) OK\n\n kernel-builder-cuda-kernels\n Build optimized CUDA kernels for PyTorch using HuggingFace kernel-builder.\n\n SKILL.md ~520 tokens\n\n baseline ████████████ 60%\n with skill ███████████████████ 95% (+35%)\n\nSaved to ./skills/kernel-builder-cuda-kernels\n```\n\nThe baseline shows how the model performs without any skill. The \"with skill\" result shows performance after the skill is injected into context. A 35% improvement means the skill is working.\n\nThe skill is saved as a directory following the [Agent Skills specification](https://agentskills.io):\n\n```\n./skills/kernel-builder-cuda-kernels/\n├── SKILL.md # Main instructions (~520 tokens)\n└── skill_meta.json # Metadata and test cases\n```\n\nOpen \\`SKILL.md\\` to see what upskill generated:\n\n```\n---\nname: kernel-builder-cuda-kernels\ndescription: Build optimized CUDA kernels for PyTorch using HuggingFace kernel-builder.\n---\n\n# Building CUDA Kernels with kernel-builder\n\n## Overview\n\nThis guide explains how to create optimized CUDA kernels for PyTorch models \nusing HuggingFace's kernel-builder. It covers project setup, kernel implementation, \nand building for specific GPU architectures like NVIDIA H100.\n\n## Project Structure\n\nproject/\n├── build.toml # Build configuration\n├── kernel_src/ # CUDA kernel implementations\n│ ├── attention.cu\n│ ├── layernorm.cu\n│ └── geglu.cu\n└── torch-ext/ # PyTorch C++ bindings\n └── torch_binding.cpp\n\n## Build Configuration\n\nCreate `build.toml` to define your kernel package:\n\n[general]\nname = \"diffuser_kernels\"\nbackends = [\"cuda\"]\n\n[general.cuda]\n# H100 is compute capability 9.0\ncapabilities = [\"9.0\"]\n\n...\n```\n\n### Evaluate on a Different Model\n\nThe important test is: does this skill help local or cheaper models to build kernels?\n\n```shell\n# Start a local OpenAI-compatible server with a web UI:\nllama-server -hf unsloth/GLM-4.7-Flash-GGUF:Q4_K_M\n\n# Evaluate on local model (llama.cpp server)\nupskill eval ./skills/my-skill/ \n --model \"unsloth/GLM-4.7-Flash-GGUF:Q4_0\" \n --base-url http://localhost:8080/v1\n```\n\n```\nGenerating skill with sonnet...\nGenerating test cases...\nEvaluating on \"unsloth/GLM-4.7-Flash-GGUF:Q4_0\"... (attempt 1)\n 40% -> 85% (+45%) OK\n\n baseline ████████░░░░░░░░░░░░ 40%\n with skill █████████████████░░░ 85% (+45%)\n\nSaved to ./skills/kernel-builder-cuda-kernels\n```\n\nA 45% improvement on `\"unsloth/GLM-4.7-Flash-GGUF:Q4_0\"` means the skill successfully transfers domain knowledge from a capable model to a faster, cheaper one. Skills that work on weaker models will definitely work on stronger ones.\n\nThis is the core value proposition: use expensive models to create skills, then deploy those skills with cheap or local models.\n\n## How the evaluation in upskill works\n\nupskill uses a teacher-student approach to evaluate models where the teacher model generates test cases for the student model to be evaluated on.\n\n1. **Teacher model** (Opus) generates the skill\n2. **Test cases** (Opus) are generated automatically from the task description\n3. **Student model** (local) is evaluated with and without the skill\n4. **Skill lift** measures the improvement\n\nIf you pass an existing skill to `upskill eval`, it will generate test cases for the skill and evaluate the model on them. Test cases are simple input/output pairs that verify the agent understands the task:\n\n```json\n{\n \"cases\": [\n {\n \"input\": \"Create a build.toml for a CUDA kernel targeting H100\",\n \"expected\": {\"contains\": \"9.0\"}\n },\n {\n \"input\": \"Write a basic CUDA kernel template with proper includes\",\n \"expected\": {\"contains\": \"cuda_runtime.h\"}\n }\n ]\n}\n```\n\nWe can also test how a skill performs across different models:\n\n```shell\nupskill eval ./skills/kernel-builder-cuda-kernels/ \n --model haiku --m kimi --runs 5\n```\n\n```\nEvaluating kernel-builder-cuda-kernels across 2 model(s)\n 3 test case(s), 5 run(s) per model\n\nhaiku\n Pass rate: 4/5 (80%) Avg assertions: 2.8/3\n\nsonnet \n Pass rate: 5/5 (100%) Avg assertions: 3.0/3\n\n┏━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓\n┃ Model ┃ Pass Rate ┃ Avg Assertions ┃ Avg Tokens ┃\n┡━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩\n│ haiku │ 4/5 │ 2.8/3 │ 1250 │\n│ kimi │ 5/5 │ 3.0/3 │ 1890 │\n└────────┴───────────┴────────────────┴────────────┘\n```\n\nThis helps you find the cost-performance sweet spot: maybe Haiku with the skill is good enough for your use case, saving significant API costs.\n\n## What's Next\n\nWe've shown that upskill can create validated skills that transfer domain expertise from powerful models to cheaper ones. The kernel-builder skill is just one example of what's possible.\n\nSome things to try:\n\n- **Generate skills for your internal tools**\n- **Build a skill library for your codebase**\n- **Capture tribal knowledge**\n- **Benchmark across models**\n\nThe approach works for any specialized task where you'd otherwise write detailed prompts repeatedly. Skills are portable across [Claude Code](https://docs.anthropic.com/en/docs/claude-code), [Codex](https://github.com/openai/codex), [Cursor](https://cursor.com/docs/context/skills), and other tools that support the [Agent Skills specification](https://agentskills.io).\n\n## Resources\n\n- [Upskill repo](https://github.com/huggingface/upskill)\n- [Agent Skills Specification](https://agentskills.io)\n- [HuggingFace kernel-builder](https://github.com/huggingface/kernels/tree/main/kernel-builder)\n", "authors": [], "captured_at": "2026-10-04T12:50:30.804479567+00:00", "category": "ai", "content_html": "

The best thing about agent skills is upskilling your agents on hard problems. There are two ways to look at that:

\n
    \n
  1. You can take Opus 4.5 or other SOTA models and tackle the hardest problems out there.
    2. \n
  2. You can take models that run on your laptop and upskill them to harder problems. In this blog post, we’ll show you how to take on the latter.
    3. \n
\n

This blog post walks through the process of using a new tool, upskill, to generate and evaluate agent skills with large models and use them with smaller models. We will benchmark upskill on the task of writing CUDA kernels for diffusers models, but the process is generally useful for cutting costs, or using smaller models on hard and domain-specific problems.

\n

What are agent skills?

\n

In case you missed it, agent skills are taking the coding agent game by storm. In fact, they’re a straightforward concept to define model context as files, like instructions as markdown and code as scripts. The file format makes them easy to generate, share, and review. In short, they’re a practical medium to share capabilities across models and tools, and they're most useful in specific domains or hard problems. Not stuff the model can do well anyway.

\n

This post showcases this process by using Claude to generate a Skill file that can be used by open source models for a complex and specialized task: write CUDA kernels. We first tried a simple skill based on existing documentation, and we found that it improved performance for some others, but not all. In fact, it could even degrade performance or increase token usage for some models. Check out the plot below to see the performance of the model with and without the basic skill.

\n

\"plot

\n

Now, let's walk through how you can use upskill to upskill your agents on hard problems, and measure performance.

\n

1. Get the teacher (Claude Opus 4.5) to build a kernel

\n

First, we use Claude Code to build a kernel interactively and export the trace. We worked through the process by instructing, validating, and adding documentation links. This somewhat naive process is important to reveal the models' initial challenges. In fact, you can iterate on this multiple times, by trying to solve the task with draft versions of the skill, and experimenting with smaller models. Each time, you can instruct the agent to improve the skill and test it on the smaller model.

\n

Here's an example of the skill that we created and have been using to build kernels. We started from this agent trace where the agent was able to build a kernel, but not without some help.

\n

2. Make an agent skill from the trace

\n

Once the teacher model has performed the task, we need them to make a skill. There are a number of effective ways to do this.

\n\n

In most cases, the first 2 options result in functional skills. However, the performance of an agent with the skill is unknown. That’s where upskill is useful, because it will also generate test cases for your skill based on the trace. It then compares the results under both scenarios: using the trace, or applying the skill. We see below that the original model (Claude Opus) met the same performance with and without the skill. This means the skill captured the task for this model. Great!

\n

\"terminal

\n

3. Take your skill to an open source, smaller, or cheaper model

\n

Finally, we need to transfer our newly created skill to the tool or model we intend to use. Most tools like codex, cursor, and opencode have settled on a consistent format for skills, which is a directory at {agent}/skills/{skill_name}/SKILL.md , so we just need to copy the skill directory to this location.

\n

With upskill we can pass a skill and a set of models to the eval command and upskill will run the test cases on those models with and without the skill to compare performance. We can see here that the skill increases accuracy on some open models, but not on all.

\n

\"performance

\n

In this case, we might want to iterate further on the gpt-oss skills by regenerating the skill. We can do upskill generate --from {skill}.

\n

There is more to agent skills than model performance. Often agents can reach a given accuracy with or without a skill, they just need to consume more tokens to get there. For recurring tasks, we want to optimize agents to use less tokens to achieve the same accuracy. The results below reveal another dimension to the skill. Some models are significantly reducing their performance token usage, whilst others are using more tokens with the skill. For example, with moonshotai/Kimi-K2-Thinking the skill is clearly effective in terms of accuracy and token usage. However, for Claude Opus 4.5 there is no clear performance increase and an increase in token usage, so you would not want to use this skill with Claude Opus 4.5.

\n

\"token

\n

tldr; try out and evaluate models with the skills you create. Use upskill eval or a similar tool to evaluate the models performance with and without skills.

\n

That’s the high level end to end of upskilling your coding agents on hard problems. Try out upskill now like this:

\n
# install upskill\npip install upskill\n\n# or use uvx \nuvx upskill --help\n\n# generate a skill based on an agent trace\nupskill generate "write nvidia kernels" --from ./trace.md\n\u000b# evaluate models on a skill\nupskill eval ./skills/my-skill/ --model haiku --model sonnet\n\n# generate skills for local models\nupskill generate "parse YAML" \n    --model opus \n    --eval-model "unsloth/GLM-4.7-Flash-GGUF:Q4_0" \n    --eval-base-url http://localhost:8080/v1\n
\n

Deep dive tutorial into building kernels with agent skills

\n

We have a high level understanding of how we can upskill an agent. Let’s now look at the use case we solved for writing CUDA kernels.

\n

We didn’t just want to write kernel code, but understand the full kernel-builder workflow: project structure, build.toml configuration, architecture-specific optimizations, and PyTorch bindings. This tutorial shows how upskill creates validated skills that actually work.

\n

The kernel-builder-cuda-kernels skill teaches Claude everything it needs to know about CUDA development: which GPU architecture to target, how to structure a kernel-builder project, when to use shared memory versus registers, and how to write PyTorch bindings.

\n

With this skill, you can tell Claude things like:

\n
Build a fused LayerNorm + GELU kernel optimized for H100.\n
\n

And Claude will create the complete project structure, CUDA implementation, and build configuration—following the exact conventions that kernel-builder expects.

\n

This isn't about generating boilerplate. The skill encodes domain expertise: H100 uses compute capability 9.0, shared memory should be aligned to 128 bytes, async memory copies require __CUDA_ARCH__ >= 900. Knowledge that would take hours to gather from documentation gets packaged into ~500 tokens that load on demand.

\n

Setup and Install

\n

Install upskill:

\n
pip install upskill\n# or use uvx for one-off runs\nuvx upskill --help\n
\n

Set your API key:

\n
export ANTHROPIC_API_KEY=sk-ant-...\nexport HF_TOKEN=hf_...\n
\n

That's it. Upskill uses Anthropic Claude Opus-4.5 model by default but also supports OpenAI and local models via OpenAI-compatible endpoints as generators. We want to use the more expensive and higher quality models to generate skills, and the smaller ones to use them. Think robin hood.

\n

Skill Generation

\n

Let's walk through generating a skill that teaches agents how to build CUDA kernels with HuggingFace's kernels library.

\n

Generate the Skill

\n

Start with a clear task description:

\n
upskill generate "build optimized CUDA kernels for PyTorch using HuggingFace kernel-builder"\n
\n

Above we used upskill, but it could in fact be any agent or chat tool and an exported trace.

\n
upskill generate "write kernels" --from <agent-trace>.md\n
\n

Also, we could start from an existing skill and add to it:

\n
upskill generate "add more error handling and edge cases" \n    --from ./skills/kernel-builder-cuda-kernels/\n
\n

upskill loads the existing skill, applies your improvements, and re-evaluates to ensure the changes help.

\n

upskill creates a skill, generates test cases, evaluates performance, and refines based on failures:

\n
Generating skill with sonnet...\nGenerating test cases...\nEvaluating on sonnet... (attempt 1)\n  60% -> 95% (+35%) OK\n\n  kernel-builder-cuda-kernels\n  Build optimized CUDA kernels for PyTorch using HuggingFace kernel-builder.\n\n  SKILL.md              ~520 tokens\n\n  baseline   ████████████                60%\n  with skill ███████████████████    95%  (+35%)\n\nSaved to ./skills/kernel-builder-cuda-kernels\n
\n

The baseline shows how the model performs without any skill. The "with skill" result shows performance after the skill is injected into context. A 35% improvement means the skill is working.

\n

The skill is saved as a directory following the Agent Skills specification:

\n
./skills/kernel-builder-cuda-kernels/\n├── SKILL.md           # Main instructions (~520 tokens)\n└── skill_meta.json    # Metadata and test cases\n
\n

Open `SKILL.md` to see what upskill generated:

\n
---\nname: kernel-builder-cuda-kernels\ndescription: Build optimized CUDA kernels for PyTorch using HuggingFace kernel-builder.\n---\n\n# Building CUDA Kernels with kernel-builder\n\n## Overview\n\nThis guide explains how to create optimized CUDA kernels for PyTorch models \nusing HuggingFace's kernel-builder. It covers project setup, kernel implementation, \nand building for specific GPU architectures like NVIDIA H100.\n\n## Project Structure\n\nproject/\n├── build.toml              # Build configuration\n├── kernel_src/             # CUDA kernel implementations\n│   ├── attention.cu\n│   ├── layernorm.cu\n│   └── geglu.cu\n└── torch-ext/              # PyTorch C++ bindings\n    └── torch_binding.cpp\n\n## Build Configuration\n\nCreate `build.toml` to define your kernel package:\n\n[general]\nname = "diffuser_kernels"\nbackends = ["cuda"]\n\n[general.cuda]\n# H100 is compute capability 9.0\ncapabilities = ["9.0"]\n\n...\n
\n

Evaluate on a Different Model

\n

The important test is: does this skill help local or cheaper models to build kernels?

\n
# Start a local OpenAI-compatible server with a web UI:\nllama-server -hf unsloth/GLM-4.7-Flash-GGUF:Q4_K_M\n\n# Evaluate on local model (llama.cpp server)\nupskill eval ./skills/my-skill/ \n    --model "unsloth/GLM-4.7-Flash-GGUF:Q4_0" \n    --base-url http://localhost:8080/v1\n
\n
Generating skill with sonnet...\nGenerating test cases...\nEvaluating on "unsloth/GLM-4.7-Flash-GGUF:Q4_0"... (attempt 1)\n  40% -> 85% (+45%) OK\n\n  baseline   ████████░░░░░░░░░░░░   40%\n  with skill █████████████████░░░   85%  (+45%)\n\nSaved to ./skills/kernel-builder-cuda-kernels\n
\n

A 45% improvement on "unsloth/GLM-4.7-Flash-GGUF:Q4_0" means the skill successfully transfers domain knowledge from a capable model to a faster, cheaper one. Skills that work on weaker models will definitely work on stronger ones.

\n

This is the core value proposition: use expensive models to create skills, then deploy those skills with cheap or local models.

\n

How the evaluation in upskill works

\n

upskill uses a teacher-student approach to evaluate models where the teacher model generates test cases for the student model to be evaluated on.

\n
    \n
  1. Teacher model (Opus) generates the skill
    2. \n
  2. Test cases (Opus) are generated automatically from the task description
    3. \n
  3. Student model (local) is evaluated with and without the skill
    4. \n
  4. Skill lift measures the improvement
    5. \n
\n

If you pass an existing skill to upskill eval, it will generate test cases for the skill and evaluate the model on them. Test cases are simple input/output pairs that verify the agent understands the task:

\n
{\n  "cases": [\n    {\n      "input": "Create a build.toml for a CUDA kernel targeting H100",\n      "expected": {"contains": "9.0"}\n    },\n    {\n      "input": "Write a basic CUDA kernel template with proper includes",\n      "expected": {"contains": "cuda_runtime.h"}\n    }\n  ]\n}\n
\n

We can also test how a skill performs across different models:

\n
upskill eval ./skills/kernel-builder-cuda-kernels/ \n    --model haiku --m kimi --runs 5\n
\n
Evaluating kernel-builder-cuda-kernels across 2 model(s)\n  3 test case(s), 5 run(s) per model\n\nhaiku\n  Pass rate: 4/5 (80%)  Avg assertions: 2.8/3\n\nsonnet  \n  Pass rate: 5/5 (100%)  Avg assertions: 3.0/3\n\n┏━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓\n┃ Model  ┃ Pass Rate ┃ Avg Assertions ┃ Avg Tokens ┃\n┡━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩\n│ haiku  │ 4/5       │ 2.8/3          │ 1250       │\n│ kimi   │ 5/5       │ 3.0/3          │ 1890       │\n└────────┴───────────┴────────────────┴────────────┘\n
\n

This helps you find the cost-performance sweet spot: maybe Haiku with the skill is good enough for your use case, saving significant API costs.

\n

What's Next

\n

We've shown that upskill can create validated skills that transfer domain expertise from powerful models to cheaper ones. The kernel-builder skill is just one example of what's possible.

\n

Some things to try:

\n\n

The approach works for any specialized task where you'd otherwise write detailed prompts repeatedly. Skills are portable across Claude Code, Codex, Cursor, and other tools that support the Agent Skills specification.

\n

Resources

\n\n", "content_markdown": "The best thing about agent skills is *upskilling* your agents on hard problems. There are two ways to look at that:\n\n1. You can take Opus 4.5 or other SOTA models and tackle the hardest problems out there.\n2. You can take models that run on your laptop and upskill them to harder problems. In this blog post, we’ll show you how to take on the latter.\n\nThis blog post walks through the process of using a new tool, `upskill`, to generate and evaluate agent skills with large models and use them with smaller models. We will benchmark `upskill` on the task of writing CUDA kernels for [`diffusers`](https://huggingface.co/docs/diffusers/en/index) models, but the process is generally useful for cutting costs, or using smaller models on hard and domain-specific problems.\n\n## What are agent skills?\n\nIn case you missed it, agent skills are taking the coding agent game by storm. In fact, they’re a straightforward concept to define model context as files, like instructions as markdown and code as scripts. The file format makes them easy to generate, share, and review. In short, they’re a practical medium to share capabilities across models and tools, and they're most useful in specific domains or hard problems. Not stuff the model can do well anyway.\n\nThis post showcases this process by using Claude to generate a Skill file that can be used by open source models for a complex and specialized task: write CUDA kernels. We first tried a simple skill based on existing documentation, and we found that it improved performance for some others, but not all. In fact, it could even degrade performance or increase token usage for some models. Check out the plot below to see the performance of the model with and without the basic skill.\n\n[![plot of model performance](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/upskill-blog/plot.png)](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/upskill-blog/plot.png)\n\nNow, let's walk through how you can use `upskill` to upskill your agents on hard problems, and measure performance.\n\n## 1. Get the teacher (Claude Opus 4.5) to build a kernel\n\nFirst, we use Claude Code to build a kernel interactively and export the trace. We worked through the process by instructing, validating, and adding documentation links. This somewhat naive process is important to reveal the models' initial challenges. In fact, you can iterate on this multiple times, by trying to solve the task with draft versions of the skill, and experimenting with smaller models. Each time, you can instruct the agent to improve the skill and test it on the smaller model.\n\nHere's an [example of the skill](https://huggingface.co/hf-skills/h100-diffusers-kernel-builder) that we created and have been using to build kernels. We started from this [agent trace](https://huggingface.co/hf-skills/h100-diffusers-kernel-builder/blob/main/agent-trace.txt) where the agent was able to build a kernel, but not without some help.\n\n## 2. Make an agent skill from the trace\n\nOnce the teacher model has performed the task, we need them to make a skill. There are a number of effective ways to do this.\n\n- Within the same session, instruct the agent to create a skill file for the task it just completed.\n- Use [Anthropic ‘skill creator’ skill](https://github.com/anthropics/skills/blob/main/skills/skill-creator/SKILL.md) either within the agent session or with an exported trace and a new agent session.\n- Use the `upskill` tool to create a skill based on the trace.\n\nIn most cases, the first 2 options result in functional skills. However, the performance of an agent with the skill is unknown. That’s where `upskill` is useful, because it will also generate test cases for your skill based on the trace. It then compares the results under both scenarios: using the trace, or applying the skill. We see below that the original model (Claude Opus) met the same performance with and without the skill. This means the skill captured the task *for this model*. Great!\n\n[![terminal evaluation](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/upskill-blog/terminal.png)](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/upskill-blog/terminal.png)\n\n## 3. Take your skill to an open source, smaller, or cheaper model\n\nFinally, we need to transfer our newly created skill to the tool or model we intend to use. Most tools like `codex`, `cursor`, and `opencode` have settled on a consistent format for skills, which is a directory at `{agent}/skills/{skill_name}/SKILL.md` , so we just need to copy the skill directory to this location.\n\nWith `upskill` we can pass a skill and a set of models to the `eval` command and `upskill` will run the test cases on those models with and without the skill to compare performance. We can see here that the skill increases accuracy on some open models, but not on all.\n\n[![performance evaluation](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/upskill-blog/accuracy.png)](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/upskill-blog/accuracy.png)\n\nIn this case, we might want to iterate further on the `gpt-oss` skills by regenerating the skill. We can do `upskill generate --from {skill}`.\n\nThere is more to agent skills than model performance. Often agents can reach a given accuracy with or without a skill, they just need to consume more tokens to get there. For recurring tasks, we want to optimize agents to use less tokens to achieve the same accuracy. The results below reveal another dimension to the skill. Some models are significantly reducing their performance token usage, whilst others are using more tokens **with** the skill. For example, with [`moonshotai/Kimi-K2-Thinking`](https://huggingface.co/moonshotai/Kimi-K2-Thinking) the skill is clearly effective in terms of accuracy and token usage. However, for Claude Opus 4.5 there is no clear performance increase and an increase in token usage, so you would not want to use this skill with Claude Opus 4.5.\n\n[![token usage](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/upskill-blog/tokens.png)](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/upskill-blog/tokens.png)\n\n**tldr;** try out and evaluate models with the skills you create. Use `upskill eval` or a similar tool to evaluate the models performance with and without skills.\n\nThat’s the high level end to end of upskilling your coding agents on hard problems. Try out upskill now like this:\n\n```shell\n# install upskill\npip install upskill\n\n# or use uvx \nuvx upskill --help\n\n# generate a skill based on an agent trace\nupskill generate \"write nvidia kernels\" --from ./trace.md\n\u000b# evaluate models on a skill\nupskill eval ./skills/my-skill/ --model haiku --model sonnet\n\n# generate skills for local models\nupskill generate \"parse YAML\" \n --model opus \n --eval-model \"unsloth/GLM-4.7-Flash-GGUF:Q4_0\" \n --eval-base-url http://localhost:8080/v1\n```\n\n## Deep dive tutorial into building kernels with agent skills\n\nWe have a high level understanding of how we can upskill an agent. Let’s now look at the use case we solved for writing CUDA kernels.\n\nWe didn’t just want to write kernel code, but understand the full kernel-builder workflow: project structure, `build.toml` configuration, architecture-specific optimizations, and PyTorch bindings. This tutorial shows how upskill creates validated skills that actually work.\n\nThe [`kernel-builder-cuda-kernels`](https://github.com/burtenshaw/kernel-skill) skill teaches Claude everything it needs to know about CUDA development: which GPU architecture to target, how to structure a kernel-builder project, when to use shared memory versus registers, and how to write PyTorch bindings.\n\nWith this skill, you can tell Claude things like:\n\n```\nBuild a fused LayerNorm + GELU kernel optimized for H100.\n```\n\nAnd Claude will create the complete project structure, CUDA implementation, and build configuration—following the exact conventions that kernel-builder expects.\n\nThis isn't about generating boilerplate. The skill encodes domain expertise: H100 uses compute capability 9.0, shared memory should be aligned to 128 bytes, async memory copies require `__CUDA_ARCH__ >= 900`. Knowledge that would take hours to gather from documentation gets packaged into ~500 tokens that load on demand.\n\n## Setup and Install\n\nInstall upskill:\n\n```shell\npip install upskill\n# or use uvx for one-off runs\nuvx upskill --help\n```\n\nSet your API key:\n\n```shell\nexport ANTHROPIC_API_KEY=sk-ant-...\nexport HF_TOKEN=hf_...\n```\n\nThat's it. Upskill uses Anthropic Claude Opus-4.5 model by default but also supports OpenAI and local models via OpenAI-compatible endpoints as generators. We want to use the more expensive and higher quality models to generate skills, and the smaller ones to use them. Think robin hood.\n\n## Skill Generation\n\nLet's walk through generating a skill that teaches agents how to build CUDA kernels with HuggingFace's [`kernels`](https://github.com/huggingface/kernels) library.\n\n### Generate the Skill\n\nStart with a clear task description:\n\n```shell\nupskill generate \"build optimized CUDA kernels for PyTorch using HuggingFace kernel-builder\"\n```\n\nAbove we used upskill, but it could in fact be any agent or chat tool and an exported trace.\n\n```shell\nupskill generate \"write kernels\" --from .md\n```\n\nAlso, we could start from an existing skill and add to it:\n\n```shell\nupskill generate \"add more error handling and edge cases\" \n --from ./skills/kernel-builder-cuda-kernels/\n```\n\nupskill loads the existing skill, applies your improvements, and re-evaluates to ensure the changes help.\n\nupskill creates a skill, generates test cases, evaluates performance, and refines based on failures:\n\n```\nGenerating skill with sonnet...\nGenerating test cases...\nEvaluating on sonnet... (attempt 1)\n 60% -> 95% (+35%) OK\n\n kernel-builder-cuda-kernels\n Build optimized CUDA kernels for PyTorch using HuggingFace kernel-builder.\n\n SKILL.md ~520 tokens\n\n baseline ████████████ 60%\n with skill ███████████████████ 95% (+35%)\n\nSaved to ./skills/kernel-builder-cuda-kernels\n```\n\nThe baseline shows how the model performs without any skill. The \"with skill\" result shows performance after the skill is injected into context. A 35% improvement means the skill is working.\n\nThe skill is saved as a directory following the [Agent Skills specification](https://agentskills.io):\n\n```\n./skills/kernel-builder-cuda-kernels/\n├── SKILL.md # Main instructions (~520 tokens)\n└── skill_meta.json # Metadata and test cases\n```\n\nOpen \\`SKILL.md\\` to see what upskill generated:\n\n```\n---\nname: kernel-builder-cuda-kernels\ndescription: Build optimized CUDA kernels for PyTorch using HuggingFace kernel-builder.\n---\n\n# Building CUDA Kernels with kernel-builder\n\n## Overview\n\nThis guide explains how to create optimized CUDA kernels for PyTorch models \nusing HuggingFace's kernel-builder. It covers project setup, kernel implementation, \nand building for specific GPU architectures like NVIDIA H100.\n\n## Project Structure\n\nproject/\n├── build.toml # Build configuration\n├── kernel_src/ # CUDA kernel implementations\n│ ├── attention.cu\n│ ├── layernorm.cu\n│ └── geglu.cu\n└── torch-ext/ # PyTorch C++ bindings\n └── torch_binding.cpp\n\n## Build Configuration\n\nCreate `build.toml` to define your kernel package:\n\n[general]\nname = \"diffuser_kernels\"\nbackends = [\"cuda\"]\n\n[general.cuda]\n# H100 is compute capability 9.0\ncapabilities = [\"9.0\"]\n\n...\n```\n\n### Evaluate on a Different Model\n\nThe important test is: does this skill help local or cheaper models to build kernels?\n\n```shell\n# Start a local OpenAI-compatible server with a web UI:\nllama-server -hf unsloth/GLM-4.7-Flash-GGUF:Q4_K_M\n\n# Evaluate on local model (llama.cpp server)\nupskill eval ./skills/my-skill/ \n --model \"unsloth/GLM-4.7-Flash-GGUF:Q4_0\" \n --base-url http://localhost:8080/v1\n```\n\n```\nGenerating skill with sonnet...\nGenerating test cases...\nEvaluating on \"unsloth/GLM-4.7-Flash-GGUF:Q4_0\"... (attempt 1)\n 40% -> 85% (+45%) OK\n\n baseline ████████░░░░░░░░░░░░ 40%\n with skill █████████████████░░░ 85% (+45%)\n\nSaved to ./skills/kernel-builder-cuda-kernels\n```\n\nA 45% improvement on `\"unsloth/GLM-4.7-Flash-GGUF:Q4_0\"` means the skill successfully transfers domain knowledge from a capable model to a faster, cheaper one. Skills that work on weaker models will definitely work on stronger ones.\n\nThis is the core value proposition: use expensive models to create skills, then deploy those skills with cheap or local models.\n\n## How the evaluation in upskill works\n\nupskill uses a teacher-student approach to evaluate models where the teacher model generates test cases for the student model to be evaluated on.\n\n1. **Teacher model** (Opus) generates the skill\n2. **Test cases** (Opus) are generated automatically from the task description\n3. **Student model** (local) is evaluated with and without the skill\n4. **Skill lift** measures the improvement\n\nIf you pass an existing skill to `upskill eval`, it will generate test cases for the skill and evaluate the model on them. Test cases are simple input/output pairs that verify the agent understands the task:\n\n```json\n{\n \"cases\": [\n {\n \"input\": \"Create a build.toml for a CUDA kernel targeting H100\",\n \"expected\": {\"contains\": \"9.0\"}\n },\n {\n \"input\": \"Write a basic CUDA kernel template with proper includes\",\n \"expected\": {\"contains\": \"cuda_runtime.h\"}\n }\n ]\n}\n```\n\nWe can also test how a skill performs across different models:\n\n```shell\nupskill eval ./skills/kernel-builder-cuda-kernels/ \n --model haiku --m kimi --runs 5\n```\n\n```\nEvaluating kernel-builder-cuda-kernels across 2 model(s)\n 3 test case(s), 5 run(s) per model\n\nhaiku\n Pass rate: 4/5 (80%) Avg assertions: 2.8/3\n\nsonnet \n Pass rate: 5/5 (100%) Avg assertions: 3.0/3\n\n┏━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓\n┃ Model ┃ Pass Rate ┃ Avg Assertions ┃ Avg Tokens ┃\n┡━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩\n│ haiku │ 4/5 │ 2.8/3 │ 1250 │\n│ kimi │ 5/5 │ 3.0/3 │ 1890 │\n└────────┴───────────┴────────────────┴────────────┘\n```\n\nThis helps you find the cost-performance sweet spot: maybe Haiku with the skill is good enough for your use case, saving significant API costs.\n\n## What's Next\n\nWe've shown that upskill can create validated skills that transfer domain expertise from powerful models to cheaper ones. The kernel-builder skill is just one example of what's possible.\n\nSome things to try:\n\n- **Generate skills for your internal tools**\n- **Build a skill library for your codebase**\n- **Capture tribal knowledge**\n- **Benchmark across models**\n\nThe approach works for any specialized task where you'd otherwise write detailed prompts repeatedly. Skills are portable across [Claude Code](https://docs.anthropic.com/en/docs/claude-code), [Codex](https://github.com/openai/codex), [Cursor](https://cursor.com/docs/context/skills), and other tools that support the [Agent Skills specification](https://agentskills.io).\n\n## Resources\n\n- [Upskill repo](https://github.com/huggingface/upskill)\n- [Agent Skills Specification](https://agentskills.io)\n- [HuggingFace kernel-builder](https://github.com/huggingface/kernels/tree/main/kernel-builder)\n", "dateCreated": "2026-10-04T12:50:30.804479567+00:00", "date_created": "2026-01-28T00:00:00+00:00", "date_modified": "2026-01-28T00:00:00+00:00", "date_published": "2026-01-28T00:00:00+00:00", "description": "The best thing about agent skills is upskilling your agents on hard problems. There are two ways to look at that: You can take Opus 4.5 or other SOTA models and tackle the hardest problems out there. You can take models that run on your…", "external_url": "https://huggingface.co/blog/upskill", "git": { "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e", "path": "items/huggingface-co/2026/01/2026-01-28-we-got-claude-to-build-cuda-kernels-and-teach-open-models.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/huggingface-co/2026/01/2026-01-28-we-got-claude-to-build-cuda-kernels-and-teach-open-models.md" }, "id": "urn:aggr:item:40db5752a2df1fab8a1adc0f577066501555995f", "isBasedOn": { "@id": "https://huggingface.co/blog/upskill" }, "mainEntity": { "@id": "https://huggingface.co/blog/upskill", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-01-28-we-got-claude-to-build-cuda-kernels-and-teach-open-models/#webpage" }, "dateModified": "2026-01-28T00:00:00+00:00", "datePublished": "2026-01-28T00:00:00+00:00", "headline": "We Got Claude to Build CUDA Kernels and teach open models!", "timeRequired": "PT10M", "url": "https://huggingface.co/blog/upskill", "wordCount": 2070 }, "name": "Archived snapshot: We Got Claude to Build CUDA Kernels and teach open models!", "preview": null, "reading_minutes": 10, "replicated_at": null, "source": "huggingface-co", "source_name": "Hugging Face - Blog", "summary": null, "tags": [], "temporalCoverage": "2026-10-04T12:50:30.804479567+00:00", "title": "We Got Claude to Build CUDA Kernels and teach open models!", "url": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-01-28-we-got-claude-to-build-cuda-kernels-and-teach-open-models/", "word_count": 2070 }