Skills are everywhere. SkillsBench counted over 47,000 unique skills across 6,300+ repositories. Everyone is writing skills. Almost nobody is testing them and most of them are AI-generated. Skills might get "vibe-checked" with a handful of manual runs, then shipped. You wouldn't ship code without tests, but why ship skills without evals?
\nThis is a practical guide to fixing that. You will learn how to define success criteria, build a lightweight eval harness, and iterate on it. I used the same steps to create and evaluate the Gemini Interactions API skill, taking it from 66.7% to 100% pass rate.
\nAgent Skills are folders of instructions, scripts, and resources that augment an agent's capabilities without retraining or fine-tuning the model. Skills follow a progressive disclosure pattern and require at minimum a SKILL.md file with:
name and description in YAML that the agent uses to decide whether to apply the skill. This is the most important piece — if it's vague, the skill won't trigger reliably.scripts/, examples/, references/ that the agent can consult during execution.Markdown
\n\n \n \nSkills fall into two categories that matter for testing:
\nBefore writing a single eval, write down what "success" means in measurable terms. Grade outcomes, not paths. Agents find creative solutions, and you don't want to penalize an unexpected route to the right answer.
\nFor our Interactions API skill, the concrete checks were: correct SDK import (from google import genai), current model IDs (not deprecated gemini-2.0-flash), uses interactions.create() not generateContent, and uses previous_interaction_id for multi-turn. Most of these are regex-checkable.
This guide assumes you already have a skill. It won't cover skill creation. Before writing any eval code, trigger the skill manually a few times. Use explicit invocations like Use the {skill name} to do X or your agent's built-in trigger (e.g. /skill {name}, $skill-name). Watch where it breaks. These first runs are not about scoring; they are about surfacing hidden assumptions. Does it assume dependencies that aren't there? Does it skip steps the user would expect?
Every fix you make during manual runs, adding a missing install command, correcting a path, tightening the description, becomes a concrete check you can automate later. Manual iteration is not wasted work.
\nFor a single skill, 10–20 prompts are enough to begin with. Each prompt tests a specific scenario and declares its own success criteria:
\nJSON
\n\nEach test declares which checks should pass via expected_checks. For agents, each test case may need its own success criteria.
I created 17 tests across five categories: core capabilities (7), guardrails for deprecated models (4), extended features not in the skill's inline examples (3), and negative controls (2). Tests span Python (12) and TypeScript (5).
\nImportant: Don't skip negative tests. A skill with too-broad description could trigger on every coding prompt.
\nTest the skill the same way the agent experiences it, e.g. through the CLI. In our case we used Gemini CLI inside a python script.
\nPython
\n\n = \n = \n = \n return \nEach check becomes a small function, using regex against the extracted code, returning a boolean.
\nPython
\n# Does the code use the correct SDK?\n\n \n return \n return \n \n# Does the code use a current model?\n = \n \n\n return not \nRegister all checks that the harness can dispatch by check_id from the prompt set, then loop through each test case:
Python
\n = \n \n\n = \n = \n \n = \n \n = \n return \nWe iterated on the Interactions API skill across ~20 test cases going from 66.7% → 100% pass rate. The two fixes that mattered most were rewriting the skill description to better match user intent (not API terminology) and replacing passive deprecation warnings with explicit instructions. The description change alone fixed 5 of 7 failures.
\nNote: We didn't use LLM-as-judge for the Interactions API skill.
\nSkills may require qualitative checks, such as code structure, naming conventions, design quality, or whether the output follows the intended patterns. These are hard to capture with regex or file existence checks alone. For example, the frontend-design skill demands "distinctive, production-grade interfaces". For such skills, you can use a second model-assisted pass.
\nUse structured output to constrain the LLM's response to a typed schema so that results become parseable and trackable.
\nPython
\n\n\n \n = \n \n\n : \n : = \n \n\n : \n : = \n : = \n : = \n : = \n : = \n \n = \n \n = \n \n = \n\n\n = \n \nUse LLM grading selectively since deterministic checks are fast, while LLM grading adds cost and latency.
\ninteractions.create()" works. "The Interactions API is the recommended approach" doesn't.If you want to go deeper, here are more resources:
\nThanks for reading! If you have any questions or feedback, please let me know on Twitter or LinkedIn.
\n", "content_markdown": "Skills are everywhere. [SkillsBench](https://arxiv.org/html/2602.12670v1) counted over **47,000 unique skills** across 6,300+ repositories. Everyone is writing skills. Almost nobody is testing them and most of them are AI-generated. Skills might get \"vibe-checked\" with a handful of manual runs, then shipped. You wouldn't ship code without tests, but why ship skills without evals?\n\nThis is a practical guide to fixing that. You will learn how to define success criteria, build a lightweight eval harness, and iterate on it. I used the same steps to create and evaluate the [Gemini Interactions API skill](https://github.com/google-gemini/gemini-skills/blob/main/skills/gemini-interactions-api/SKILL.md), taking it from 66.7% to 100% pass rate.\n\n## What Are Agent Skills?\n\n[Agent Skills](https://agentskills.io/home) are folders of instructions, scripts, and resources that augment an agent's capabilities without retraining or fine-tuning the model. Skills follow a **progressive disclosure** pattern and require at minimum a `SKILL.md` file with:\n\n1. **Frontmatter** (trigger): A `name` and `description` in YAML that the agent uses to decide whether to apply the skill. This is the most important piece — if it's vague, the skill won't trigger reliably.\n2. **Body** (instructions): Markdown guidance for how to perform the task: what APIs to use, what patterns to follow, what to avoid.\n3. **Resources** (optional): `scripts/`, `examples/`, `references/` that the agent can consult during execution.\n\nMarkdown\n\n```md\n---\nname: gemini-interactions-api\ndescription: Use this skill when writing code that calls the Gemini API\n for text generation, multi-turn chat, image generation, streaming responses,\n function calling, structured output, or migrating from generateContent SDK.\n---\n \n# Gemini Interactions API Skill\n \nThe Interactions API is a unified interface for interacting with Gemini models\nand agents...\n```\n\nSkills fall into two categories that matter for testing:\n\n- **Capability** skills help the agent do something the base model can't do consistently. These may become unnecessary as models improve; evals will tell you when that's happened.\n- **Preference** skills document specific workflows. These are durable, but only as valuable as their fidelity to your actual workflow, and evals verify that fidelity.\n\n## Define Success Before You Write the Skill\n\nBefore writing a single eval, write down what \"success\" means in measurable terms. Grade outcomes, not paths. Agents find creative solutions, and you don't want to penalize an unexpected route to the right answer.\n\n- **Outcome:** Did the skill produce a usable result? Code compiles, the image rendered, the document got created, the API returned a valid response. This is the baseline. If the output doesn't work, nothing else matters.\n- **Style & Instructions:** Does the output follow your conventions and the skill's directives? Right SDK, correct model IDs, team's naming conventions, the formatting you specified.\n- **Efficiency:** How much time, tokens, and effort did it take? No unnecessary retries, reasonable token count, no command thrashing. This is the most undervalued dimension. Two runs can produce identical correct output, but one burned 3x the tokens. Regressions here are real costs that compound.\n\nFor our Interactions API skill, the concrete checks were: correct SDK import (`from google import genai`), current model IDs (not deprecated `gemini-2.0-flash`), uses `interactions.create()` not `generateContent`, and uses `previous_interaction_id` for multi-turn. Most of these are regex-checkable.\n\n## Practical Guide to an Eval Harness\n\nThis guide assumes you already have a skill. It won't cover skill creation. Before writing any eval code, trigger the skill manually a few times. Use explicit invocations like `Use the {skill name} to do X` or your agent's built-in trigger (e.g. `/skill {name}`, `$skill-name`). Watch where it breaks. These first runs are not about scoring; they are about surfacing hidden assumptions. Does it assume dependencies that aren't there? Does it skip steps the user would expect?\n\nEvery fix you make during manual runs, adding a missing install command, correcting a path, tightening the description, becomes a concrete check you can automate later. Manual iteration is not wasted work.\n\n### 1\\. Create a prompt set\n\nFor a single skill, **10–20 prompts** are enough to begin with. Each prompt tests a specific scenario and declares its own success criteria:\n\nJSON\n\n```json\n[\n {\n \"id\": \"py_basic_generation\",\n \"prompt\": \"Write a Python script that sends a text prompt to Gemini and prints the response.\",\n \"language\": \"python\",\n \"should_trigger\": true,\n \"expected_checks\": [\"correct_sdk\", \"no_old_sdk\", \"current_model\", \"interactions_api\"]\n },\n {\n \"id\": \"py_deprecated_model\",\n \"prompt\": \"Write a Python script using Gemini 2.0 Flash with the Interactions API.\",\n \"language\": \"python\",\n \"should_trigger\": true,\n \"expected_checks\": [\"correct_sdk\", \"interactions_api\", \"deprecated_model_rejected\"]\n },\n {\n \"id\": \"negative_unrelated\",\n \"prompt\": \"Write a Python script that reads a CSV and plots a bar chart using matplotlib.\",\n \"language\": \"python\",\n \"should_trigger\": false,\n \"expected_checks\": []\n }\n]\n```\n\nEach test declares which checks should pass via `expected_checks`. For agents, [each test case may need its own success criteria](https://blog.langchain.com/evaluating-deep-agents-our-learnings/).\n\nI created 17 tests across five categories: core capabilities (7), guardrails for deprecated models (4), extended features not in the skill's inline examples (3), and negative controls (2). Tests span Python (12) and TypeScript (5).\n\n**Important**: Don't skip negative tests. A skill with too-broad description could trigger on every coding prompt.\n\n### 2\\. Run the agent and capture output\n\nTest the skill the same way the agent experiences it, e.g. through the CLI. In our case we used Gemini CLI inside a python script.\n\nPython\n\n```python\ndef run_gemini_cli(prompt):\n cmd = [\n \"gemini\",\n \"-m\", \"gemini-3-flash-preview\",\n \"--output-format\", \"json\",\n \"--yolo\",\n \"-p\", prompt,\n ]\n result = subprocess.run(cmd, capture_output=True, text=True, timeout=3600)\n data = json.loads(result.stdout.strip())\n return CLIOutput(\n response_text=data.get(\"response\", \"\"),\n stats=data.get(\"stats\", {}),\n exit_code=result.returncode,\n )\n```\n\n### 3\\. Write deterministic checks\n\nEach check becomes a small function, using regex against the extracted code, returning a boolean.\n\nPython\n\n```python\n# Does the code use the correct SDK?\ndef check_correct_sdk(code, language):\n if language == \"python\":\n return bool(re.search(r\"from\\s+google\\s+import\\s+genai\", code))\n return bool(re.search(r\"\"\"['\"]@google/genai['\"]\"\"\", code))\n \n# Does the code use a current model?\nDEPRECATED_MODELS = [\"gemini-2.0-flash\", \"gemini-1.5-pro\", \"gemini-1.5-flash\"]\n \ndef check_current_model(code, language):\n return not any(model in code for model in DEPRECATED_MODELS)\n```\n\nRegister all checks that the harness can dispatch by `check_id` from the prompt set, then loop through each test case:\n\nPython\n\n```python\nCHECK_REGISTRY = {\n \"correct_sdk\": check_correct_sdk,\n \"current_model\": check_current_model,\n \"interactions_api\": check_interactions_api,\n \"no_old_sdk\": check_no_old_sdk,\n # ... 11 checks total\n}\n \ndef run_eval(test_case):\n output = run_gemini_cli(test_case[\"prompt\"])\n code = extract_code_blocks(output.response_text)\n \n results = {}\n for check_id in test_case[\"expected_checks\"]:\n results[check_id] = CHECK_REGISTRY[check_id](code, test_case[\"language\"])\n return results\n```\n\nWe iterated on the Interactions API skill across ~20 test cases going from **66.7% → 100%** pass rate. The two fixes that mattered most were rewriting the skill description to better match user intent (not API terminology) and replacing passive deprecation warnings with explicit instructions. The description change alone fixed 5 of 7 failures.\n\n### 4\\. Add LLM-as-judge for qualitative checks\n\n*Note: We didn't use LLM-as-judge for the Interactions API skill.*\n\nSkills may require qualitative checks, such as code structure, naming conventions, design quality, or whether the output follows the intended patterns. These are hard to capture with regex or file existence checks alone. For example, the [frontend-design skill](https://github.com/anthropics/claude-code/tree/main/plugins/frontend-design/skills/frontend-design) demands \"distinctive, production-grade interfaces\". For such skills, you can use a second model-assisted pass.\n\nUse structured output to constrain the LLM's response to a typed schema so that results become parseable and trackable.\n\nPython\n\n```python\nfrom google import genai\nfrom pydantic import BaseModel, Field\n \nclient = genai.Client()\n \nclass CheckResult(BaseModel):\n passed: bool\n notes: str = Field(description=\"Brief explanation of the assessment.\")\n \nclass DesignEvalResult(BaseModel):\n overall_pass: bool\n score: int = Field(ge=0, le=100)\n typography: CheckResult = Field(\n description=\"Uses distinctive fonts, avoids generic choices like Inter/Arial/Roboto.\")\n color_cohesion: CheckResult = Field(\n description=\"Cohesive palette with CSS variables, no timid evenly-distributed colors.\")\n layout: CheckResult = Field(\n description=\"Intentional spatial composition — asymmetry, overlap, or bold grid choices.\")\n generic_ai_avoidance: CheckResult = Field(\n description=\"No purple-gradient-on-white, no cookie-cutter patterns.\")\n \ncode_to_grade = open(\"./output/landing-page.html\").read()\n \ninteraction = client.interactions.create(\n model=\"gemini-3-flash-preview\",\n input=f\"\"\"Evaluate this frontend code against these requirements:\n- Has distinctive, production-grade interfaces.\n- No generic fonts (Inter, Roboto, Arial),\n- No clichéd color schemes (purple gradients on white), and predictable layouts.\n \n Grade each dimension and provide an overall score.\n \n Code:\n {code_to_grade}\"\"\",\n response_format=DesignEvalResult.model_json_schema(),\n)\n \nresult = DesignEvalResult.model_validate_json(interaction.outputs[-1].text)\nprint(f\"Score: {result.score}/100\")\nfor name in [\"typography\", \"color_cohesion\", \"layout\", \"generic_ai_avoidance\"]:\n check = getattr(result, name)\n print(f\" {name}: {'✓' if check.passed else '✗'} — {check.notes}\")\n```\n\nUse LLM grading selectively since deterministic checks are fast, while LLM grading adds cost and latency.\n\n## Best Practices for Evaluating Agent Skills\n\n1. **Start with the skill description.** The name and description are the trigger mechanism. A vague description means the skill won't be used when it should, or when it shouldn't.\n2. **Use directives, not information.** Models follow instructions better than they infer implications. \"Always use `interactions.create()`\" works. \"The Interactions API is the recommended approach\" doesn't.\n3. **Include negative tests.** Add prompts where the skill should *not* trigger. A skill with too-broad keywords will be used on every request.\n4. **Start small, extend from failures.** Begin with 10–20 prompts drawn from real usage. Every user-reported wrong output becomes a new test case.\n5. **Grade outcomes, not paths.** Agents find creative solutions. Don't penalize an unexpected route to the right answer.\n6. **Isolate each run.** Use a clean environment for every test case. Accumulated context can bleed between test cases and masks real failures.\n7. **Run multiple trials.** Agent behavior is nondeterministic. A single pass/fail per prompt isn't enough signal, run 3–5 trials per case and look at the distribution, not just one result.\n8. **Test across harnesses.** The same skill can behave differently depending on the agent framework running it. If your skill is used across tools, eval it in each one.\n9. **Graduate your evals.** Capability evals start at a low pass rate and give you a hill to climb. Once they hit ~100%, they graduate into regression evals that protect against backsliding.\n10. **Detect skill retirement.** Run your evals with the skill unloaded. If they still pass, the model has absorbed the skill's value. Retire it.\n\n## Other Resources\n\nIf you want to go deeper, here are more resources:\n\n- [Demystifying Evals for AI Agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)\n- [Improving Skill-Creator](https://claude.com/blog/improving-skill-creator-test-measure-and-refine-agent-skills)\n- [Testing Agent Skills Systematically with Evals](https://developers.openai.com/blog/eval-skills)\n- [Evaluating Deep Agents](https://blog.langchain.com/evaluating-deep-agents-our-learnings/)\n- [SkillsBench](https://arxiv.org/html/2602.12670v1)\n- [Skills for Evaluating AI Products](https://hamel.dev/blog/posts/evals-skills/)\n\n* * *\n\nThanks for reading! If you have any questions or feedback, please let me know on [Twitter](https://twitter.com/_philschmid) or [LinkedIn](https://www.linkedin.com/in/philipp-schmid-a6a2bb196/).\n", "dateCreated": "2026-09-27T19:29:15.293681927+00:00", "date_created": "2026-03-04T00:00:00+00:00", "date_modified": "2026-03-04T00:00:00+00:00", "date_published": "2026-03-04T00:00:00+00:00", "description": "Learn how to systematically test and improve agent skills using deterministic checks and a real-world Gemini API example.", "external_url": "https://www.philschmid.de/testing-skills", "git": { "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e", "path": "items/philschmid-de/2026/03/2026-03-04-practical-guide-to-evaluating-and-testing-agent-skills.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/philschmid-de/2026/03/2026-03-04-practical-guide-to-evaluating-and-testing-agent-skills.md" }, "id": "urn:aggr:item:4ac5bcba6db4cc67572fc22a381dac06848c4d4b", "isBasedOn": { "@id": "https://www.philschmid.de/testing-skills" }, "mainEntity": { "@id": "https://www.philschmid.de/testing-skills", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/philschmid-de/2026-03-04-practical-guide-to-evaluating-and-testing-agent-skills/#webpage" }, "dateModified": "2026-03-04T00:00:00+00:00", "datePublished": "2026-03-04T00:00:00+00:00", "headline": "Practical Guide to Evaluating and Testing Agent Skills", "timeRequired": "PT8M", "url": "https://www.philschmid.de/testing-skills", "wordCount": 1659 }, "name": "Archived snapshot: Practical Guide to Evaluating and Testing Agent Skills", "preview": null, "reading_minutes": 8, "replicated_at": null, "source": "philschmid-de", "source_name": "philschmid.de - RSS feed", "summary": "Learn how to systematically test and improve agent skills using deterministic checks and a real-world Gemini API example.", "tags": [], "temporalCoverage": "2026-09-27T19:29:15.293681927+00:00", "title": "Practical Guide to Evaluating and Testing Agent Skills", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2026-03-04-practical-guide-to-evaluating-and-testing-agent-skills/", "word_count": 1659 }