{ "@context": "https://schema.org", "@id": "https://aggr.aymericbeaumet.com/items/hamel-dev/2026-03-02-evals-skills-for-coding-agents/#webpage", "@type": [ "WebPage", "ArchiveComponent" ], "articleBody": "![](https://hamel.dev/blog/posts/evals-skills/cover-original.png)\n\nToday, Shreya Shankar and I are publishing [evals skills](https://github.com/ai-evals-course/evals-skills), a set of skills for AI product evals[^1].\n\nEval tools often get in the way. They nudge you toward [generic off-the-shelf metrics](https://hamel.dev/blog/posts/evals-faq/index.html#q-should-i-use-ready-to-use-evaluation-metrics) and [fully automated evals](https://parlance-labs.com/blog/posts/auto-evals.html) before you’ve looked at your data. These skills help you avoid common mistakes we’ve seen helping 50+ companies and teaching students in our [AI Evals course](https://maven.com/parlance-labs/evals).\n\nThere are many [easily avoidable footguns](https://hamel.dev/blog/posts/revenge/index.html) in evals. These skills help you avoid them.\n\n**evals-start** is the entry point. It looks at your situation and routes you to the right skill. Most of the time it will send you to one of these two:\n\n- **eval-audit**, if you already have an eval pipeline. It inspects your setup and recommends next steps. [^2]\n- **error-discovery**, if you have traces but haven’t analyzed them yet. It builds a customized annotation interface and helps you sample traces intelligently. Shreya does a live walkthrough of using this skill [here](https://youtu.be/tqUDjc1HzO4).\n\n## The skills\n\nInstall the skills:\n\n```\nnpx skills add https://github.com/ai-evals-course/evals-skills\n```\n\nThen give your agent this prompt:\n\n> Run the evals-start skill from the evals plugin and follow the skill it picks. If it picks eval-audit, investigate each diagnostic area using a separate subagent in parallel, then synthesize the findings into a single report.\n\nIf you’re experienced with evals, skip the router and pick the skill you need:\n\n| Skill | What it does |\n| ----------------------- | ----------------------------------------------------------------------------------------- |\n| evals-start | Entry point. Routes to the skill that matches your situation |\n| eval-audit | Audit an eval pipeline and surface problems with prioritized severity |\n| error-discovery | Build a review app, select diverse samples, and organize your notes into failure modes |\n| generate-synthetic-data | Create diverse synthetic test inputs using dimension-based tuple generation |\n| write-judge-prompt | Design LLM-as-Judge evaluators for subjective quality criteria |\n| validate-evaluator | Calibrate LLM judges against human labels using data splits, TPR/TNR, and bias correction |\n| evaluate-rag | Evaluate retrieval and generation quality in RAG pipelines |\n| build-review-interface | Build custom annotation interfaces for human trace review |\n\nThese skills are only a starting point. To make them better, tune them to be more specific to your data and domain.\n\nThe repo is [ai-evals-course/evals-skills](https://github.com/ai-evals-course/evals-skills).\n\nYou can find me on [X](https://x.com/hamelhusain) or email me through my [newsletter](https://ai.hamel.dev).\n\n[^1]: Not foundation model benchmarks like MMLU or HELM that measure general LLM capabilities. Product evals measure whether *your* pipeline works on *your* task with *your* data. If you aren’t familiar with product-specific AI evals, check out my [AI Evals FAQ](https://hamel.dev/blog/posts/evals-faq/index.html).\n\n[^2]: The audit isn’t a complete solution, but it will catch common problems.\n", "authors": [ "Hamel Husain", "Shreya Shankar" ], "captured_at": "2026-09-27T19:29:15.293681927+00:00", "category": "programming", "content_html": "

\"\"

\n

Today, Shreya Shankar and I are publishing evals skills, a set of skills for AI product evals1.

\n

Eval tools often get in the way. They nudge you toward generic off-the-shelf metrics and fully automated evals before you’ve looked at your data. These skills help you avoid common mistakes we’ve seen helping 50+ companies and teaching students in our AI Evals course.

\n

There are many easily avoidable footguns in evals. These skills help you avoid them.

\n

evals-start is the entry point. It looks at your situation and routes you to the right skill. Most of the time it will send you to one of these two:

\n\n

The skills

\n

Install the skills:

\n
npx skills add https://github.com/ai-evals-course/evals-skills\n
\n

Then give your agent this prompt:

\n
\n

Run the evals-start skill from the evals plugin and follow the skill it picks. If it picks eval-audit, investigate each diagnostic area using a separate subagent in parallel, then synthesize the findings into a single report.

\n
\n

If you’re experienced with evals, skip the router and pick the skill you need:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
SkillWhat it does
evals-startEntry point. Routes to the skill that matches your situation
eval-auditAudit an eval pipeline and surface problems with prioritized severity
error-discoveryBuild a review app, select diverse samples, and organize your notes into failure modes
generate-synthetic-dataCreate diverse synthetic test inputs using dimension-based tuple generation
write-judge-promptDesign LLM-as-Judge evaluators for subjective quality criteria
validate-evaluatorCalibrate LLM judges against human labels using data splits, TPR/TNR, and bias correction
evaluate-ragEvaluate retrieval and generation quality in RAG pipelines
build-review-interfaceBuild custom annotation interfaces for human trace review
\n

These skills are only a starting point. To make them better, tune them to be more specific to your data and domain.

\n

The repo is ai-evals-course/evals-skills.

\n

You can find me on X or email me through my newsletter.

\n
\n
    \n
  1. \n

    Not foundation model benchmarks like MMLU or HELM that measure general LLM capabilities. Product evals measure whether your pipeline works on your task with your data. If you aren’t familiar with product-specific AI evals, check out my AI Evals FAQ. ↩

    \n
    2. \n
  2. \n

    The audit isn’t a complete solution, but it will catch common problems. ↩

    \n
    3. \n
\n
\n", "content_markdown": "![](https://hamel.dev/blog/posts/evals-skills/cover-original.png)\n\nToday, Shreya Shankar and I are publishing [evals skills](https://github.com/ai-evals-course/evals-skills), a set of skills for AI product evals[^1].\n\nEval tools often get in the way. They nudge you toward [generic off-the-shelf metrics](https://hamel.dev/blog/posts/evals-faq/index.html#q-should-i-use-ready-to-use-evaluation-metrics) and [fully automated evals](https://parlance-labs.com/blog/posts/auto-evals.html) before you’ve looked at your data. These skills help you avoid common mistakes we’ve seen helping 50+ companies and teaching students in our [AI Evals course](https://maven.com/parlance-labs/evals).\n\nThere are many [easily avoidable footguns](https://hamel.dev/blog/posts/revenge/index.html) in evals. These skills help you avoid them.\n\n**evals-start** is the entry point. It looks at your situation and routes you to the right skill. Most of the time it will send you to one of these two:\n\n- **eval-audit**, if you already have an eval pipeline. It inspects your setup and recommends next steps. [^2]\n- **error-discovery**, if you have traces but haven’t analyzed them yet. It builds a customized annotation interface and helps you sample traces intelligently. Shreya does a live walkthrough of using this skill [here](https://youtu.be/tqUDjc1HzO4).\n\n## The skills\n\nInstall the skills:\n\n```\nnpx skills add https://github.com/ai-evals-course/evals-skills\n```\n\nThen give your agent this prompt:\n\n> Run the evals-start skill from the evals plugin and follow the skill it picks. If it picks eval-audit, investigate each diagnostic area using a separate subagent in parallel, then synthesize the findings into a single report.\n\nIf you’re experienced with evals, skip the router and pick the skill you need:\n\n| Skill | What it does |\n| ----------------------- | ----------------------------------------------------------------------------------------- |\n| evals-start | Entry point. Routes to the skill that matches your situation |\n| eval-audit | Audit an eval pipeline and surface problems with prioritized severity |\n| error-discovery | Build a review app, select diverse samples, and organize your notes into failure modes |\n| generate-synthetic-data | Create diverse synthetic test inputs using dimension-based tuple generation |\n| write-judge-prompt | Design LLM-as-Judge evaluators for subjective quality criteria |\n| validate-evaluator | Calibrate LLM judges against human labels using data splits, TPR/TNR, and bias correction |\n| evaluate-rag | Evaluate retrieval and generation quality in RAG pipelines |\n| build-review-interface | Build custom annotation interfaces for human trace review |\n\nThese skills are only a starting point. To make them better, tune them to be more specific to your data and domain.\n\nThe repo is [ai-evals-course/evals-skills](https://github.com/ai-evals-course/evals-skills).\n\nYou can find me on [X](https://x.com/hamelhusain) or email me through my [newsletter](https://ai.hamel.dev).\n\n[^1]: Not foundation model benchmarks like MMLU or HELM that measure general LLM capabilities. Product evals measure whether *your* pipeline works on *your* task with *your* data. If you aren’t familiar with product-specific AI evals, check out my [AI Evals FAQ](https://hamel.dev/blog/posts/evals-faq/index.html).\n\n[^2]: The audit isn’t a complete solution, but it will catch common problems.\n", "dateCreated": "2026-09-27T19:29:15.293681927+00:00", "date_created": "2026-03-02T08:00:00+00:00", "date_modified": "2026-03-02T08:00:00+00:00", "date_published": "2026-03-02T08:00:00+00:00", "description": "Today, Shreya Shankar and I are publishing evals skills, a set of skills for AI product evals1. Eval tools often get in the way. They nudge you toward generic off-the-shelf metrics and fully automated evals before you’ve looked at your…", "external_url": "https://hamel.dev/blog/posts/evals-skills/", "git": { "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e", "path": "items/hamel-dev/2026/03/2026-03-02-evals-skills-for-coding-agents.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/hamel-dev/2026/03/2026-03-02-evals-skills-for-coding-agents.md" }, "id": "urn:aggr:item:febe8bb9a9279695c7b9cfc445558d8ef0b5bbc4", "isBasedOn": { "@id": "https://hamel.dev/blog/posts/evals-skills/" }, "mainEntity": { "@id": "https://hamel.dev/blog/posts/evals-skills/", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/hamel-dev/2026-03-02-evals-skills-for-coding-agents/#webpage" }, "author": [ { "@type": "Person", "name": "Hamel Husain" }, { "@type": "Person", "name": "Shreya Shankar" } ], "dateModified": "2026-03-02T08:00:00+00:00", "datePublished": "2026-03-02T08:00:00+00:00", "headline": "Evals Skills for Coding Agents", "timeRequired": "PT2M", "url": "https://hamel.dev/blog/posts/evals-skills/", "wordCount": 433 }, "name": "Archived snapshot: Evals Skills for Coding Agents", "preview": null, "reading_minutes": 2, "replicated_at": null, "source": "hamel-dev", "source_name": "Hamel's Blog", "summary": "Today, Shreya Shankar and I are publishing evals skills, a set of skills for AI product evals1. Eval tools often get in the way. They nudge you toward generic off-the-shelf metrics and fully automated evals before you’ve looked at your data. These skills help you avoid common mistakes we’ve seen helping 50+ companies and teaching students in our AI Evals course. Why skills for evals There are many easily avoidable footguns in evals. These skills help you avoid them. evals-start is the entry point. It looks at your situation and routes you to the right skill. Most of the time it will send you to one of these two: eval-audit, if you already have an eval pipeline. It inspects your setup and recommends next steps. 2 error-discovery, if you have traces but haven’t analyzed them yet. It builds a customized annotation interface and helps you sample traces intelligently. Shreya does a live walkthrough of using this skill here. The skills Install the skills: npx skills add https://github.com/ai-evals-course/evals-skills Then give your agent this prompt: Run the evals-start skill from the evals plugin and follow the skill it picks. If it picks eval-audit, investigate each diagnostic area using a separate subagent in parallel, then synthesize the findings into a single report. If you’re experienced with evals, skip the router and pick the skill you need: Skill What it does evals-start Entry point. Routes to the skill that matches your situation eval-audit Audit an eval pipeline and surface problems with prioritized severity error-discovery Build a review app, select diverse samples, and organize your notes into failure modes generate-synthetic-data Create diverse synthetic test inputs using dimension-based tuple generation write-judge-prompt Design LLM-as-Judge evaluators for subjective quality criteria validate-evaluator Calibrate LLM judges against human labels using data splits, TPR/TNR, and bias correction evaluate-rag Evaluate retrieval and generation quality in RAG pipelines build-review-interface Build custom annotation interfaces for human trace review These skills are only a starting point. To make them better, tune them to be more specific to your data and domain. The repo is ai-evals-course/evals-skills. You can find me on X or email me through my newsletter. Footnotes Not foundation model benchmarks like MMLU or HELM that measure general LLM capabilities. Product evals measure whether your pipeline works on your task with your data. If you aren’t familiar with product-specific AI evals, check out my AI Evals FAQ.↩︎ The audit isn’t a complete solution, but it will catch common problems.↩︎", "tags": [ "ai", "evals" ], "temporalCoverage": "2026-09-27T19:29:15.293681927+00:00", "title": "Evals Skills for Coding Agents", "url": "https://aggr.aymericbeaumet.com/items/hamel-dev/2026-03-02-evals-skills-for-coding-agents/", "word_count": 433 }