![\"A](\"https://hamel.dev/blog/posts/claude-auto-evals/images/choose-eval-annotated.png\")
Decision models have exploded onto the scene recently, starting with TypeSafe’s Jev, followed by Perplexity’s Decider, Liquid’s d1, Cloudflare’s Clef, and OpenAI’s Decisions API1. These models promise fast, inexpensive classification.
\nDecision models are an appealing low-cost alternative to LLM judges2 for automated evals. But whether they’re the right choice depends on the cost, performance, and latency requirements of your use case.
\nWe’ve also noticed people using the default decision threshold of 0.50 without checking whether it works well for their task. This can leave a lot of performance on the table.
\nIn this post, we’ll start by showing how to tune thresholds to get the best results from decision models. With those thresholds tuned, we’ll compare cost and performance to help you assess which model fits your use case. Finally, we’ll explore how to interpret model scores when deciding whether to act, and how calibration can make them more useful as confidence estimates.
\nWe used 100 traces, or logs of conversations and actions, from an apartment leasing agent. The task is to identify traces containing an error, such as a failed booking or a missed handoff to a human.3 We used the same dataset and broad failure-finding task in our prior post on automated evals. Here, each decision model returns a failure probability for each trace.
\nWe tested Jev 1.13, Perplexity Decider 1.1 27B, Liquid d1 and d1-ultra, Cloudflare Clef Flash, and OpenAI Decisions (GPT-6 Luna)1. Each model received the same evaluation criteria. The appendix shows the prompt and the structured version used for Jev.
\nWe compare models on two dimensions:
\nFor brevity, we leave latency aside because it depends on factors outside this experiment’s control.5
\nWe assign each trace one of two labels, pass or fail. This is a binary pass/fail eval, which we recommend because it encourages clear criteria and is easier to align with human preferences.
\nEach decision model returns a score from 0 to 1 that estimates how likely a trace is to contain a failure. The APIs turn this score into a decision using a default decision threshold of 0.50. Traces with scores at or above the threshold are flagged as failures. Using this default without checking it against your data can give you subpar results.
\nFor example, a model could assign every failure a score of 0.40 and every passing trace a score of 0.10. Even though this model classifies traces perfectly at a threshold of 0.30, the default threshold of 0.50 would catch nothing.
\nIn practice, scores for failures and passing traces will overlap. Lowering the threshold catches more failures but can also raise more false alarms. Choosing a threshold means balancing:
\nThe right tradeoff depends on your business goals. For example, criminal trials set a high bar for conviction to avoid convicting innocent people, emphasizing precision. Cancer screening may emphasize recall, accepting more false alarms to miss fewer cases.6 For this comparison, we’ll give precision and recall equal weight using the F1 score, their harmonic mean. F1 ranges from 0 to 1, and higher is better.
\nThe curves below show how each model scores failures (orange) and passing traces (blue). Taller parts of a curve mean more scores from that class fall in that range. Scores at or above the dashed threshold line are flagged. Use the toggle to compare 0.50 with Optimized for best F1, which chooses a threshold for each model.7
\nOpenAI separates the two classes best on this task, which is also reflected in its higher F1 score.
\nIt’s also important not to blindly maximize F1 without thinking about your minimum requirements for the classifier to be useful. For example, if this classifier is used for an automated alerting system, I would want a minimum of 80% on both precision and recall so that I’m minimizing false alarms while catching a sufficient number of failures.
\nThe chart below shows how precision and recall change at different thresholds.8 The shaded region marks where both reach at least 80%.
\nPerplexity, Liquid d1-ultra, and OpenAI Decisions meet our requirements, but we still need to choose a decision threshold within the shaded region for each model. In the next section, we’ll discuss how to tune those thresholds.
\nFor this example, we’ll optimize F1. For our candidates, we’ll restrict tuning to thresholds that meet both 80% minimums on the training data whenever possible. We leave the other models unconstrained to illustrate the effect of tuning.9
\nWith labeled traces and saved model scores, you can ask a coding agent to tune the thresholds. Below is an example prompt you can modify for your use case:
\n\n\nUsing my saved model scores and binary labels (1 means failure), tune a decision threshold to maximize F1 while requiring at least 80% precision and 80% recall. Use stratified five-fold cross-validation, choosing thresholds on the training examples and measuring performance on the remaining examples. If comparing models, use the same splits for each. Only select thresholds that meet both minimums on the training examples. If none qualifies, report that rather than relaxing either requirement. Report validation precision, recall, and F1, along with any splits where no threshold qualified or validation performance fell below my minimums. Compare with the default threshold of 0.50. Reuse the saved scores without making new model calls.
\n
Here’s how tuning for F1 changed each model’s performance. We group the three candidates from the precision-recall chart together and show the other models in gray to illustrate how much tuning can help.
\nOur candidate models gained less from F1 tuning than Liquid d1 and Clef Flash, which fell below our minimums. However, even a small shift in precision and recall can affect business costs or the user experience, so choose the tradeoff deliberately. In practice, it’s unlikely that precision and recall are equally important (as we assumed in this post with F1). For example, the F-beta score lets you give precision more weight with beta below 1, or recall more weight with beta above 1. Optimizing a different metric could lead to different thresholds and results.
\nThe chart below compares each model’s F1 (after tuning) against cost, focused on our remaining candidates:
\nThe dashed line shows the Pareto frontier among our remaining candidates. For any given F1 score, it identifies the cheapest candidate that meets or exceeds that score. This particular chart is a bit silly as there are only three options, but it’s a useful illustration for how to think about cost vs. performance.
\nPerplexity was the cheapest of these candidates at 0.82 cents per 100 eval runs, with an F1 of 0.79. OpenAI Decisions reached an F1 of 0.87 at 3.72 cents per 100 eval runs, about 4.5 times the cost.
\nBeyond pass/fail evals, you may want to use a decision model’s score as a confidence estimate. But can you take that number literally? For a score of 0.80 to mean an 80% chance of failure, roughly 80% of traces with similar scores should contain failures.
\nThe chart below shows each model’s probability calibration: how closely its predicted probabilities match observed outcomes.10 Each point represents about 20 traces with similar scores. A point on the diagonal means the predicted probability matches the failure rate in that group. Above the line, the model underestimates the risk of failure. Below it, the model overestimates that risk.
\nIf the predicted probabilities don’t match the observed outcomes, we can adjust them using another model. The second panel shows one method, Platt scaling, which fits a simple curve to make that adjustment.11
\nPerplexity, Liquid d1, and Clef Flash move closer to the diagonal after calibration. For example, one group of Clef Flash traces had an average predicted failure probability of about 8%, but 60% contained failures.
\nJev and Liquid d1-ultra were already well calibrated on this dataset and needed little adjustment.
\nPlatt scaling isn’t always the best fit. For example, OpenAI’s probabilities got slightly worse with the version of Platt scaling we used.12 You can direct your coding agent to perform calibration with the following prompt:
\n\n\nUsing my saved failure probabilities and binary labels (1 means failure), test whether Platt scaling improves each model’s probabilities. Use stratified five-fold cross-validation with the same splits for every model. Fit the calibration mapping on each training split and apply it to the remaining traces. Fit on log-odds, clipping probabilities away from 0 and 1 first. Plot predicted probabilities against observed failure rates before and after calibration. Report Brier scores (lower is better), and explain which models improved or got worse. Reuse the saved scores without making new model calls.
\n
We kept the eval task in this post broad to make the differences between models easier to see. In practice, we recommend building a separate judge for each failure mode. With a narrower task and more specific guidance, we expect the performance gaps between models to shrink.
\nI like to start with a larger model that has acceptable performance on a task and then try smaller models to see if they can match the larger model’s performance for less cost. Prompt tuning can also be helpful if you have high-quality labels.13 I discuss how to craft a good judge prompt in this post.
\nFor this particular dataset, OpenAI had breakout performance at a higher cost. Perplexity and Liquid d1-ultra offer lower-cost options with lower performance. We have observed that model performance varies substantially across tasks and datasets, which means it’s important to measure things yourself.
\nTo recap:
\nThis process is iterative. You may improve the prompt, then need to tune the threshold again or revisit calibration. In this post, we focused on threshold tuning and calibration because we’ve seen many people use decision models off the shelf without considering these factors.
\nYou might also be wondering how decision models stack up to LLMs. OpenAI Decisions uses GPT-6 Luna, which delivered our highest F1 at the highest cost, suggesting that harder eval tasks may justify paying more for an LLM. In a broader comparison, Randy Olson found that decision models could match LLMs on classification and routing at lower cost, while LLMs pulled ahead on tasks requiring knowledge and reasoning.
\nNone of the ideas discussed in this post are new. Choosing a precision-recall tradeoff, tuning thresholds and comparing performance vs. cost are part of evaluating any classifier. To learn more about these fundamentals, I recommend Machine Learning with Python Cookbook, 2nd Edition by Kyle Gallatin and Chris Albon.
\nFinally, if you’re a data scientist or ML engineer wondering where you belong in the AI age, these are skills you already have. It’s time to get revenge 🙂.
\nPerplexity, Liquid d1 and d1-ultra, Clef Flash, and OpenAI Decisions received the prompt below. Jev received equivalent criteria in a structured prompt, shown in the example below. The scores and labels used in the charts are available as JSON.
\nDecide whether `messages` contains at least one substantive failure, risk, or quality problem in the AI assistant's behavior or tool use. This is one complete production trace from an AI that handles apartment-leasing customer support over phone or messaging. The trace includes the conversation plus any tool calls and tool outputs.\n\nEvaluate the AI assistant and its tool use, not the customer. Treat system messages inside `messages` as the instructions governing the assistant's behavior and allowed capabilities. Inspect the full trace, including final turns and all tool calls and outputs. Base the judgment only on evidence visible in this trace. Do not assume external policies, channel behavior, or business requirements that are not present. Do not assume every trace failed. A short, ambiguous, or inconclusive trace is not an issue unless visible evidence establishes a substantive failure.\n\nReturn the probability that at least one substantive issue is present; the issue does not need to be named or clustered.\n\nCount as an issue when supported by the trace: the assistant contradicts an applicable system instruction or tool result; makes an unsupported factual or operational claim; misses, misuses, repeats, or falsely claims completion of a required tool action, transfer, booking, cancellation, follow-up, or confirmation; materially misunderstands or leaves an explicit user request unresolved; or exposes internal actions, placeholders, or broken content to the customer in a way that materially harms the interaction.\n\nDo not count as an issue: a minor stylistic preference with no meaningful effect; a failure caused only by the customer's behavior; internal tool metadata that was not exposed to the customer and did not cause an incorrect response; a hypothetical concern requiring information outside the trace; or insufficient evidence and inconclusive conversations.\nNoteExample: A structured prompt
\nThis is how we structured the prompt for Jev’s requests, with separate fields for instructions and evaluation criteria.
\n\nVersions tested: Jev 1.13 (announced September 15, 2026), Perplexity Decider 1.1 27B, Liquid d1 and d1-ultra, Cloudflare Clef Flash, and OpenAI Decisions. OpenAI Decisions uses GPT-6 Luna. The API returns typed answers, including a probability from 0 to 1 for a true/false question. This is a separate API from Structured Outputs, which generates responses matching a custom JSON schema. We include it because it serves the same classification task. ↩ ↩2
\nAn LLM judge uses a prompt to evaluate an AI system’s output or behavior. We recommend asking it for a binary pass/fail decision about a specific failure mode. This encourages clear criteria and makes it easier to compare the judge’s decisions with human labels. ↩
\nWe scored 98 production traces after excluding two internal QA examples from the 100-trace dataset. The frozen reference contains 61 known failures and 37 traces without a labeled failure: 39 failures from the original human review plus 22 discoveries from Braintrust and Arize that we manually validated. Models received masked traces without human labels or reviewer notes. Perplexity, Liquid, Clef Flash, and OpenAI received the same instructions and serialized inputs; Jev received equivalent criteria through its structured API. The original conversations are private. ↩
\nAPI cost covers 100 eval runs, one per trace. We use Liquid’s prices as of October 17, 2026, $0.02 per million input tokens for d1 and $0.042 for d1-ultra. The experiment recorded d1 at $0.04; we have repriced its saved token usage. Other models use the rates in the experiment’s operational metrics. Performance excludes two internal QA traces. Threshold tuning uses saved scores and requires no additional model calls. ↩
\nLatency depends on how you batch or parallelize requests, your arrangement with the vendor, and your distance from its infrastructure, among other factors. Batching and concurrency can also change cost and throughput. See TypeSafe’s multi-question requests and client performance guidance. Its launch discussion also notes the location of its latency tests relative to its service. Mean response times were 4.33 seconds for Jev, 7.85 for Perplexity, 7.38 for Liquid d1, 1.56 for Liquid d1-ultra, 2.09 for Clef Flash, and 2.77 for OpenAI. These came from separate runs and do not establish comparative throughput or tail latency. ↩
\nThese examples illustrate how the consequences of mistakes affect the tradeoff. In U.S. criminal trials, proof beyond a reasonable doubt sets a high bar for conviction. In cancer screening, a missed case can delay care, while a false alarm can lead to anxiety and unnecessary procedures. A positive screening result usually requires further testing. The NCI explains both risks; the appropriate balance depends on the screening test and population. ↩
\nThe density plots show unconstrained F1 tuning, before we introduce minimum precision and recall requirements. The later comparison charts apply those requirements to candidate models where feasible. All use the same five-fold cross-validation splits. Thresholds were chosen on training traces, and precision, recall, and F1 were measured on separate traces. We average those measurements across repeated splits. In the optimized density plots, the dashed line shows the average selected threshold. The displayed scores come from the individual splits, rather than applying that average threshold to the full dataset. The density curves use Gaussian smoothing with bandwidth 0.07 and reflection at 0 and 1. Each class has an area of one. Smoothing affects only the display. ↩
\nA precision-recall curve plots precision against recall as the decision threshold changes. It shows the tradeoffs available for a model across thresholds. Scikit-learn’s precision-recall guide explains how to read and plot these curves. ↩
\nWe identify candidates from the full-sample precision-recall curves. In each training split, we choose the highest-F1 threshold among those reaching at least 80% precision and 80% recall. When no threshold qualifies, we use the unconstrained F1 optimum and record the split as infeasible. This occurred in 36 of 100 training splits for Perplexity and 29 for Liquid d1-ultra, and never for OpenAI. Non-candidates always use unconstrained F1 tuning. We keep all validation predictions, including those from fallback splits, in the reported averages. The minimums apply during selection and do not guarantee the same performance on new examples. The threshold-selection audit records each split without trace IDs. ↩
\nA calibrated failure probability of 0.80 means that, among many examples receiving scores near 0.80, roughly 80% contain failures. Platt scaling and isotonic regression use labeled examples to learn an adjustment from model scores to probabilities. Platt scaling fits a sigmoid curve. Isotonic regression fits a more flexible mapping that preserves score order, but is more prone to overfitting with small datasets. Neither guarantees improvement on new examples. Scikit-learn’s probability calibration guide explains both methods and how to assess them. ↩
\nThe reliability chart groups each model’s scores into five groups of 19 or 20 traces, sorted by probability. The adjusted scores are each trace’s mean out-of-fold probability from the saved cross-validation results. We fit Platt scaling to the log-odds of the original probabilities. Each mapping is fitted without the traces it predicts. The chart summarizes this small sample and does not establish an exact failure rate for a future trace. The plotted calibration results are available as JSON. ↩
\nOn the saved out-of-fold probability summaries, Jev’s Brier score was 0.187 before adjustment and 0.188 afterward. Liquid d1-ultra’s was 0.173 and 0.172, while OpenAI’s was 0.159 and 0.164. Brier score measures average squared probability error, so lower is better. It reflects discrimination as well as calibration. These results describe the log-odds Platt fit shown in the chart. An exploratory refit directly on probabilities, using the same folds, gave OpenAI a Brier score of 0.154. This illustrates sensitivity to the fitting method. It does not establish which method will work best on new data. ↩
\nGEPA uses feedback from model runs to propose prompt revisions and test them against labeled examples. It retains candidates with different strengths, so a prompt that helps a difficult example can remain in consideration. Incorrect labels can steer this search toward the wrong changes. Keep a separate test set outside prompt development for the final check. In our small manual Jev experiment, F1 at the default threshold rose from 0.78 with a simple prompt to 0.80 with a checklist of failure types. That experiment did not use GEPA or establish how much systematic optimization would help. Our FAQ on choosing an eval model discusses prompt optimization and model comparison. ↩
\nAnthropic released new eval tooling for Claude Code. Their claude-api plugin now includes a new build_eval and hill-climb command that helps you build evals, check the graders, and improve your application against them.
I usually don’t review eval tools. Software changes so often that a review has a short shelf life. But a first-party tool from Anthropic is likely to influence how people approach evals, so I wanted to try it.
\nIsaac Flath and I livestreamed ourselves using it on conversation traces from an apartment leasing assistant. Here’s what we found:
\nClaude started by suggesting several potential failures, then asked us to pick one straight away to turn into an eval. It gave us the below menu of options, with call-transfer rules as the recommended choice. We hadn’t yet reviewed the conversations ourselves, so it was hard to know if this was a real failure or worth prioritizing. Despite this, we went ahead with the recommended choice, because we figured that’s what typical users would do.
\n
I believe you should be looking at data first to inform your understanding and prioritize which evals to write. An agent can help you find issues, but you still should do error analysis to decide which failures deserve attention before proceeding.
\nNext, Claude created Markdown files for looking at data associated with the call-transfer failure. In the screenshot below, Claude asks us to skim inputs.md and “tell it” which labels are wrong. That meant reading long conversations in an editor and reporting corrections separately in a chat.
![\"Red](\"https://hamel.dev/blog/posts/claude-auto-evals/images/markdown-review-annotated-v3.png\")
We found this very silly as we were using a coding agent, so it should have built an annotation app that made the conversations easy to read and let us leave feedback in-situ. We eventually asked Claude to build a web app for us and used that instead.
\nLater in the workflow, Claude made an initial attempt at creating an evaluator for the call-transfer failure. It presented aggregate label counts and asked us, “Would you have scored any case differently?” without giving us enough information to know if the labels were correct. A recurring theme of the workflow was to jump too fast into creating artifacts or asking us for approval without helping us understand the data.
\n![\"Red](\"https://hamel.dev/blog/posts/claude-auto-evals/images/score-review-question-annotated-v3.png\")
Next, the tool created a call-transfer evaluator that checked four different failures at once:
\nThere were too many things bundled into this evaluator. I would prefer to scope the eval to focus on one error at a time, or at the very least separate the evals into those that needed a code-based eval vs a LLM as a Judge.
\nClaude’s description of the evaluator was also confusing:
\n\n\nprotocol_ok is the headline. A case passes only if all four checks pass. On “should not transfer” calls, protocol_ok is 1 if no transfer happened.
\n
This AI slop is hard to read. I’d much rather see the code or the judge prompt so I can understand whats being created. I’ve found that it always pays to read the prompt, especially for something as important as an eval. Below is a screenshot of what this part of the workflow looked like:
\n![\"Claude](\"https://hamel.dev/blog/posts/claude-auto-evals/images/bundled-grader.png\")
I was impressed by this plugin’s out-of-the-box ability to discover issues that other auto-eval approaches haven’t been able to find! It found issues with human handoff, formatting, voice agents, and more. It’s still better to look at your data iteratively with an agent, but this was the strongest performance I’ve seen with a more “one-shot” issue discovery approach.
\nAnthropic’s blog post introducing this tool broadly conveys thinking that I agree with, such as the importance of looking at data, sampling intelligently, not saturating your own evals, etc. I’m really happy more people are thinking about evals this way.
\nI’d hold off for now. I’d want the workflow to help me explore the data before committing to an evaluator, with a better review interface from the start. Additionally, I’m already quite happy with what coding agents can do using these Eval skills Shreya and I put together which is less opinionated (but more flexible).
\nI’ve since spoken with the author of the Claude eval plugin. He was appreciative of the feedback and said he’d update the plugin accordingly, so I expect it to change soon. It could be worth revisiting in the future.
\nEven as this plugin changes, I hope this walkthrough helps you assess other eval tools. Make sure the tool helps you understand your data before choosing evals and inspect its judgments carefully. I believe much of the eval workflow should happen in a web application rather than chat to remove friction from data exploration and annotation.
\nRemember, if an eval tool doesn’t put looking at data at the center of your workflow, it’s not worth using.
\nThanks to Isaac Flath for reviewing this article and joining me for the livestream.
\n", "date_modified": "2026-09-30T07:00:00+00:00", "date_published": "2026-09-30T07:00:00+00:00", "external_url": "https://hamel.dev/blog/posts/claude-auto-evals/", "id": "urn:aggr:item:e62d33ebcaf86175438d343218786d6d06766807", "summary": "Anthropic released new eval tooling for Claude Code. Their claude-api plugin now includes a new build_eval and hill-climb command that helps you build evals, check the graders, and improve your application against them. I usually don’t…", "tags": [ "programming", "ai", "evals" ], "title": "Claude’s new auto eval tool", "url": "https://aggr.aymericbeaumet.com/items/hamel-dev/2026-09-30-claude-s-new-auto-eval-tool/" }, { "authors": [ { "name": "Hamel Husain" }, { "name": "Shreya Shankar" } ], "content_html": "This document curates the most common questions Shreya and I received while teaching 5,000+ engineers and PMs AI Evals. Warning: These are sharp opinions about what works in most cases. They are not universal truths. Use your judgment.
\nBrowse the questions that interest you, or choose a guide below for a curated reading path through the FAQs and related articles.
\n| Where are you? | \nThis sounds like me | \n
|---|---|
| I’m new to evals | \nI’ve heard the term, but I’m not sure what evals involve or whether I need them. | \n
| I don’t know what to test | \nI’m building an AI product, but I haven’t figured out which failures to measure or what good performance looks like. | \n
| I don’t trust my eval scores | \nWe have evals, but the scores don’t match our judgment of the outputs, or tests pass while users still encounter problems. | \n
| My product feels too hard to evaluate | \nOur outputs are subjective, long, or involve many steps. Even a knowledgeable person has trouble deciding whether they’re right. | \n
| Evals take too much time or money | \nWe’re spending too much effort reviewing outputs, maintaining tests, or running evaluators. | \n
Browse all questions by section.
\nAI evals are tests that tell you whether an AI system is doing what you want. They give your team feedback when the product drifts from user needs or business goals. The failures they catch also become data you can use to improve the system.
\nMore formally, evaluation is the systematic measurement of quality. Each eval checks one behavior on relevant examples and returns a score or structured review. Most AI products need several evals because they can fail in different ways.
\nWhen you hear the word “evals,” it usually refers to one of two things: model benchmarks or product evals.
\nModel benchmarks compare general-purpose models on shared tasks. Model providers publish these benchmark results when they release new models. Common examples include GPQA Diamond for graduate-level science reasoning, Terminal-Bench for agents doing complex work in command-line environments, and MMLU for knowledge and reasoning across a wide range of subjects. These scores can help you choose a promising model as a starting point. To assess quality on your own tasks you need product evals, which we discuss next.
\nProduct evals measure whether your specific AI product does what you want it to do. They turn your judgment about what a good product experience looks like into metrics you can track. Product evals encompass all components of your product, including the model, prompts, retrieval, tools, and application code. This flavor of evals are focused on capturing failures that matter to users and the business.
\nConsider an order-cancellation agent. Its product evals might check whether it selected the correct order and waited for the cancellation tool to succeed before telling the user the order was canceled. A high score on GPQA Diamond or Terminal-Bench gives you little information on this, because those benchmarks don’t have access to your systems.
\nThere are several mechanisms you can use to implement product evals, including code assertions, human review, LLM judges, and online experiments. The right method depends on the failure being measured, and is discussed in greater detail in this series.
\nIn the rest of the AI Evals FAQ, we focus on product evals. It starts with analyzing traces to discover real failure modes. We then turn important failures into targeted evals and use the results to guide changes. Finally, rerunning the evals tells us whether the system improved.
\nIf you are completely new to product-specific evals, see these posts:
\n| \n | Guide | \nWhat it covers | \n
|---|---|---|
![\"Your](\"https://hamel.dev/blog/posts/evals/images/diagram-cover.webp\") | \nPart 1: Your AI Product Needs Evals | \nBuild a domain-specific evaluation system with scoped tests, trace review, human evaluation, and experiments. | \n
![\"Using](\"https://hamel.dev/blog/posts/llm-judge/images/cover_img.webp\") | \nPart 2: Using LLM-as-a-Judge For Evaluation: A Complete Guide | \nCapture a domain expert’s judgment, automate it with an LLM judge, and validate the judge against human labels. | \n
![\"A](\"https://hamel.dev/blog/posts/field-guide/images/field_guide_2.webp\") | \nPart 3: A Field Guide to Rapidly Improving AI Products | \nUse error analysis, realistic data, and trustworthy evals to run a sustained product-improvement loop. | \n
A trace is the complete record of all actions, messages, tool calls, and data retrievals from a single initial user query through to the final response. It includes every step across all agents, tools, and system components in a session: multiple user messages, assistant responses, retrieved documents, and intermediate tool interactions.
\nNote on terminology: Different observability vendors use varying definitions of traces and spans. Alex Strick van Linschoten’s analysis highlights these differences (screenshot below):
\n![\"\"](\"https://hamel.dev/blog/posts/evals-faq/alex.webp\")
Vendor differences in trace definitions as of 2025-07-02
\n\nStart with error analysis, not infrastructure. Spend 30 minutes manually reviewing 20-50 LLM outputs whenever you make significant changes. Use one domain expert who understands your users as your quality decision maker (a “benevolent dictator”).
\nUse a notebook to review traces and analyze data, or build your own custom annotation interface with an AI coding assistant like Claude or Codex. Either way, you can write arbitrary code, visualize data, and iterate quickly. The video below shows a simple annotation interface built inside a notebook.
\n\nIt’s important to recognize that evaluation is part of the development process rather than a distinct line item, similar to how debugging is part of software development.
\nYou should always be doing error analysis. When you discover issues through error analysis, many will be straightforward bugs you’ll fix immediately. These fixes don’t require separate evaluation infrastructure as they’re just part of development.
\nThe decision to build automated evaluators comes down to cost-benefit analysis. If you can catch an error with a simple assertion or regex check, the cost is minimal and probably worth it. But if you need to align an LLM-as-judge evaluator, consider whether the failure mode warrants that investment.
\nIn the projects we’ve worked on, we’ve spent 60-80% of our development time on error analysis and evaluation. Expect most of your effort to go toward understanding failures (i.e. looking at data) rather than building automated checks.
\nBe wary of optimizing for high eval pass rates. If you’re passing 100% of your evals, you’re likely not challenging your system enough. A 70% pass rate might indicate a more meaningful evaluation that’s actually stress-testing your application. Focus on evals that help you catch real issues, not ones that make your metrics look good.
\n\nYes. Even with perfect models, you still need to verify they’re solving the right problem. The need for systematic error analysis, domain-specific testing, and monitoring will still be important.
\nToday’s prompt engineering tricks might become obsolete, but you’ll still need to understand failure modes. Additionally, a LLM cannot read your mind, and research shows that people need to observe the LLM’s behavior in order to properly externalize their requirements.
\nFor deeper perspective on this debate, see these two viewpoints: “The model is the product” versus “The model is NOT the product”.
\n\nDon’t try to sell your team on “evals”. Instead, show them what you find when you look at the data.
\nStart by doing the error analysis yourself. Look at 50 to 100 real user conversations and find the most common ways the product is failing. Use these findings to tell a story with data.
\nPresent your team with:
\nFrame evaluation as part of development, not optional testing. Keep a running log of the errors you catch, what you learned, the fix, and the likely impact you avoided. Share it weekly or monthly. A concrete report such as “we caught 47 issues before users saw them” makes the value easier to see than an abstract pitch about evals.
\nThis approach builds trust. Don’t just show dashboards and metrics; tell the story of what you’re finding in the data. By narrating your findings, you teach the team what you’re learning, providing immediate value. When you fix an issue, show how the error rate for that specific problem went down. Soon, your team will see the progress and ask how you’re doing it. Let results instead of methods lead the conversation.
\nThis is similar to classic machine learning projects, where outcomes are speculative and progress is bounded by iterating on experiments. In this situation, it’s important that you share the learnings from each experiment to show progress and encourage investment.
\n\nError analysis is the most important activity in evals. Error analysis helps you decide what evals to write in the first place. It allows you to identify failure modes unique to your application and data. The process involves:
\nGathering representative traces of user interactions with the LLM. If you do not have any data, you can generate synthetic data to get started.
\nHuman annotator(s) (ideally a benevolent dictator) review and write open-ended notes about traces, noting any issues. This process is akin to “journaling” and is adapted from qualitative research methodologies. Start by annotating at least 30 traces yourself before reviewing suggestions from an agent. When beginning, it is recommended to focus on noting the first failure observed in a trace, as upstream errors can cause downstream issues, though you can also tag all independent failures if feasible. A domain expert should be performing this step.
\nCategorize the open-ended notes into a “failure taxonomy.” In other words, group similar failures into distinct categories. Axial coding is the most important step. At the end, count the number of failures in each category. You can use an LLM to help with this step.
\nHave your agent cluster the data and choose a diverse initial sample. After your first 30 annotations, let it search the remaining traces for likely instances of the failures you described. Accept or reject its suggestions and keep iterating until you reach theoretical saturation, meaning new reviews stop revealing failure modes or changing existing ones.
\nA working pool of roughly 100 diverse traces is a useful guardrail for this human-agent loop. The agent can focus your attention on the most informative traces, so you no longer have to read all 100 sequentially. See how many examples you need for each kind of eval for the full breakdown.
\nYou should frequently revisit this process. There are advanced ways to sample data more efficiently, like clustering, sorting by user feedback, and sorting by high probability failure patterns. Over time, you’ll develop a “nose” for where to look for failures in your data.
\nDo not skip error analysis. It ensures that the evaluation metrics you develop are supported by real application behaviors instead of counter-productive generic metrics (which most platforms nudge you to use). For examples of how error analysis can be helpful, see this video, or this blog post.
\nHere is a visualization of the error analysis process by one of our students, Pawel Huryn - including how it fits into the overall evaluation process:
\n![\"Infographic](\"https://hamel.dev/blog/posts/evals-faq/pawel-error-analysis.webp\")
No. Writing a rubric before you review examples can get in the way.
\nLet’s get some definitions out of the way:
\nBoth can help, but treat your initial expectations as a starting point that you will revise.
\nIt’s often better to wait until you’ve reviewed some examples before developing a detailed rubric. Reviewers can become so focused on checking each item that they overlook problems outside the rubric. It’s important to give reviewers room to notice things you didn’t anticipate. This change in what you consider good is called “criteria drift”.
\nFor example, let’s say you have a support agent that handles refunds and it escalates refunds to a human per your policy. You might only realize that the process is frustrating for the user after reading a few interactions. Don’t underestimate the degree of criteria drift that will happen as you review examples!
\nWe recommend using error analysis to systematically review examples and decide what might belong in the rubric. This involves writing open-ended notes about what looks wrong, then group similar notes to see which problems recur. See this live demo for a walkthrough.
\nAfter doing error analysis, you can write a better rubric informed by user and application behavior. You should periodically do error analysis to make sure your rubric is current.
\n\nYes. When reviewing interactions, write down anything that makes the product less useful. This includes missing or broken features that have nothing to do with the model. Additionally, don’t focus on why the error occurred, as that should only come after you prioritize which issues to fix.
\nFor example, a support agent might tell a customer that an order has shipped without providing a tracking link. Even if your AI doesn’t have the ability to fetch a tracking link, record that problem. A prerequisite to building evals is to identify and prioritize which issues to fix through error analysis. Some of these issues may end up being engineering or design issues that don’t need an automated evaluator, but they are still important to fix!
\nLastly, we’ve found that deferring root-cause analysis and focusing on problems allows you to write higher-quality annotations while looking at more data.
\n\nBuilding evals is a pipeline, and each stage needs a different amount of data. We describe these stages below:
\n| Stage | \nWhat to do | \n
|---|---|
| 1. Review the application | \nRead traces and write down the ways your application fails. This process is called error discovery. Start with 100 diverse traces and annotate at least the first 30 yourself. | \n
| 2. Create and validate evaluators | \nChoose between two evaluator types. Use a code-based eval when an objective rule can identify the failure. Include Pass and Fail examples for every condition and important edge case. Use an LLM judge when the failure requires human judgment. Label 100 to 200 examples for each failure mode. | \n
| 3. Build a repeatable eval set | \nCollect examples that represent important workflows and confirmed failures. Run this set when you change your application. These sets often grow to 100 or more examples. | \n
A trace is a complete record of one user session with your application. Ask a coding agent to help you sample the initial pool so it covers different users and workflows. Our evals plugin can help with sampling and build an annotation interface for your traces.
\nWe recommend annotating at least 30 traces with a process called error discovery yourself before asking the agent to suggest failures. Write free-text notes about anything that seems wrong from the user’s perspective. These examples give the agent a concrete record of your judgment.
\nKeep this first pass manual. If the agent starts suggesting problems too early, its guesses can bias your judgment. You may miss failures that depend on product context or your definition of a good user experience.
\nAfter 30 traces, ask the agent to search the remaining pool for similar examples. Review every suggestion yourself. Accept or reject each one and correct the agent when it misunderstands your criteria.
\nContinue until new traces stop revealing failure modes or changing existing ones. Qualitative researchers call this theoretical saturation. We recommend reviewing at least 100 traces. Continue past 100 while you are still learning.
\nIf you want to see the process done live, watch the live walkthrough. The video shows how Shreya Shankar uses an agent to review traces quickly while keeping a human in charge of the failure criteria.
\nThis review produces a failure taxonomy, which is a list of the specific ways your application fails. Use that taxonomy to decide which evaluators to build.
\nChoose an evaluator for each important failure mode. The evaluator type determines how many labeled examples you need. Code-based evals work for objective rules. LLM judges work for failures that require human judgment.
\nUse a code-based eval when a deterministic rule can identify the failure. Examples include checking whether JSON parses or whether a tool call uses the correct arguments.
\nThe number of examples depends on the scenarios the check covers. At minimum, include examples that should Pass and Fail for every condition. Add important edge cases you found during error discovery. A check with one rule may need only a few examples that Pass and a few that Fail.
\nUse an LLM judge when the failure requires subjective or domain-specific judgment. Plan to label 100 to 200 examples for each failure mode. Reuse labeled traces from error discovery when they match the failure mode, then collect more until you reach that range. The labels should come from a trusted domain expert and contain enough Pass and Fail examples to evaluate both classes.
\nSplit these examples into train, dev, and test sets. Use 10 to 20 percent for train examples that may appear in the prompt. Use 40 to 45 percent for dev while refining the judge. Reserve the remaining 40 to 45 percent for one final test. When possible, include 30 to 50 Pass examples and 30 to 50 Fail examples in both the dev and test sets.
\nThe validation guide explains the full process. The judge-validation flashcard is a good visual reference as well.
\nAfter validating the evaluators, assemble the examples you will run repeatedly during development.
\nStart with examples from error discovery that capture important failure modes. Add confirmed failures as you find them.
\nA purpose-built eval set often grows to 100 or more examples. Coverage determines the final size. Each important workflow and known failure should be represented, and the set should remain cheap enough to run often. Code-based checks and LLM judges can run over the same examples. The CI evals FAQ explains how to use this set during development.
\n\nWhile user feedback is a good way to narrow in on problematic traces, other methods are also useful. Here are three complementary approaches:
\nThe simplest approach is reviewing a random sample of traces. If you find few issues, escalate to stress testing: create queries that deliberately test your prompt constraints to see if the AI follows your rules.
\nUse existing evals to find problematic traces and potential issues. Once you’ve identified these, you can proceed with the typical evaluation process starting with error analysis.
\nFor more sophisticated trace discovery, use outlier detection, metric-based sorting, and stratified sampling to find interesting traces. Generic metrics can serve as exploration signals to identify traces worth reviewing, even if they don’t directly measure quality.
\n\nRe-run error analysis when making significant changes: new features, prompt updates, model switches, or major bug fixes. A useful heuristic is to set a goal for reviewing at least 100+ fresh traces each review cycle. Typical review cycles we’ve seen range from 2-4 weeks. See this FAQ on how to sample traces effectively.
\nBetween major analyses, review 10-20 traces weekly, focusing on outliers: unusually long conversations, sessions with multiple retries, or traces flagged by automated monitoring. Adjust frequency based on system stability and usage growth. New systems need weekly analysis until failure patterns stabilize. Mature systems might need only monthly analysis unless usage patterns change. Always analyze after incidents, user complaint spikes, or metric drift. Scaling usage introduces new edge cases.
\n\nEval datasets naturally get stale as your product and users change. Use regular error analysis to find new problems and update your examples or reference answers. How often you review depends on your use case and how quickly your product or usage changes.
\nLike unit tests, evals can catch problems that return after a fix. However, evals often cost considerably more than unit tests to maintain and run. Therefore, you should weigh each eval’s cost against the value of its signals. If everything keeps passing, this is a sign that the eval is less useful and should be run less often or retired. You should phase out expensive evals like LLM-as-a-judge more agressively than cheaper evals.
\nAs your eval set changes, its scores may no longer be directly comparable with older scores. That is ok! One purpose of evals are to provide you with challenges you can hill climb against. These challenges should change as your product evolves to help you keep improving.
\nFor tracking progress with metrics over longer time horizons, it’s often better to use product metrics in addition to evals. Examples of product metrics include: churn, active users, revenue, etc.
\n\nA common mistake is prompting an LLM to "give me test queries" without structure, resulting in generic, repetitive outputs. A structured approach using dimensions produces far better synthetic data for testing LLM applications.
Use synthetic data to start error analysis before you have enough production traffic, or to test a known failure that appears rarely in real data. Define the variation you need, generate examples, run them through the full system, and review the resulting traces.
\nSynthetic data cannot tell you how common a failure is in production. It can also miss details that matter in specialized domains. Compare synthetic examples with real data as soon as real data becomes available. See when synthetic data may be unreliable for cases that require extra review.
\nStart by defining dimensions: categories that describe different aspects of user queries. Each dimension captures one type of variation in user behavior. For example:
\nStart with failure hypotheses. If you lack intuition about failure modes, use your application extensively or recruit friends to use it. Then choose dimensions targeting those likely failures.
\nCreate tuples manually first: Write 20 tuples by hand. Each tuple selects one value from each dimension. Example: (Vegan, Italian, Multi-step). This manual work helps you understand your problem space.
\nScale with two-step generation:
\nThis separation avoids repetitive phrasing. The (Vegan, Italian, Multi-step) tuple becomes: "I need a dairy-free lasagna recipe that I can prep the day before."
You can generate tuples two ways:
\nCross product then filter: Generate all dimension combinations, then filter with an LLM. Guarantees coverage including edge cases. Use when most combinations are valid.
\nDirect LLM generation: Ask the LLM to generate tuples directly. This produces more realistic combinations, but it tends toward generic outputs and misses rare scenarios. Use it when many dimension combinations are invalid.
\nFix obvious problems first: Don’t generate synthetic data for issues you can fix immediately. If your prompt doesn’t mention dietary restrictions, fix the prompt rather than generating specialized test queries.
\nAfter iterating on your tuples and prompts, run these synthetic queries through your actual system to capture full traces. A pool of roughly 100 diverse traces is a useful starting point for failure discovery. Have an agent help with sampling, annotate at least 30 traces yourself, then review the agent’s suggestions until your learning plateaus. See how many examples you need for error discovery for the full explanation.
\nHere is a visual that helps visualize the process.
\n\nYes: synthetic data can mislead or mask issues. For guidance on generating synthetic data when appropriate, see What is the best approach for generating synthetic data?
\nCommon scenarios where synthetic data fails:
\nComplex domain-specific content: LLMs often miss the structure, nuance, or quirks of specialized documents (e.g., legal filings, medical records, technical forms). Without real examples, critical edge cases are missed.
\nLow-resource languages or dialects: For low-resource languages or dialects, LLM-generated samples are often unrealistic. Evaluations based on them won’t reflect actual performance.
\nWhen validation is impossible: If you can’t verify synthetic sample realism (due to domain complexity or lack of ground truth), real data is important for accurate evaluation.
\nHigh-stakes domains: In high-stakes domains (medicine, law, emergency response), synthetic data often lacks subtlety and edge cases. Errors here have serious consequences, and manual validation is difficult.
\nUnderrepresented user groups: For underrepresented user groups, LLMs may misrepresent context, values, or challenges. Synthetic data can reinforce biases in the training data of the LLM.
\nThere is no replacement for looking at real interactions. This situation is not ideal, but there are some things you can do. Here are some options, in order of preference:
\nTry to find real data you are allowed to inspect. A customer may agree to share a subset of traces, or test users may let you review their interactions. Even limited access gives you examples of how people use the product.
\nIf you cannot inspect the data yourself, work with domain experts who are allowed to see it. Make your product easier for them to verify as part of their normal work. For example, a medical research assistant could show a clinician the evidence behind each claim and flag conflicting sources for review. The clinician can correct a specific claim or resolve a conflict while using the product. Those decisions can provide additional data for evals, subject to the same restrictions on what you can store and share.
\nTo design this well, learn how the experts check an answer. Give them links to the supporting evidence and smaller pieces of work they can review. Asking whether the final answer was helpful often tells you too little about what went wrong. I discuss this approach in this post.
\nRedact or edit traces so they can be shared. If sensitive information cannot be stored, redact it before logging. Redaction tools can miss sensitive information, so check their output. When edited traces can be shared, removing personal information and changing sensitive details can make real examples usable for review. Check that those edits preserve the behavior you need to evaluate.
\nIf none of the above options are possible, synthetic data should be your last resort. Synthetic data can help you find initial problems but has the downside that it only gives you limited evidence about how real users will behave. Read more about when synthetic data may be unreliable.
\n\n\nComplex applications often support vastly different query patterns—from “What’s the return policy?” to “Compare pricing trends across regions for products matching these criteria.” Each query type exercises different system capabilities, leading to confusion on how to design eval criteria.
\n
Error Analysis is all you need. Your evaluation strategy should emerge from observed failure patterns (e.g. error analysis), not predetermined query classifications. Rather than creating a massive evaluation matrix covering every query type you can imagine, let your system’s actual behavior guide where you invest evaluation effort.
\nDuring error analysis, you’ll likely discover that certain query categories share failure patterns. For instance, all queries requiring temporal reasoning might struggle regardless of whether they’re simple lookups or complex aggregations. Similarly, queries that need to combine information from multiple sources might fail in consistent ways. These patterns discovered through error analysis should drive your evaluation priorities. It could be that query category is a fine way to group failures, but you don’t know that until you’ve analyzed your data.
\nTo see an example of basic error analysis in action, see this video.
\n\nThere are many ways to sample production traces for review. Here are some common methods.
\n| Method | \nWhat it does | \nMain limitation | \n
|---|---|---|
| Random | \nSelects traces with equal probability. | \nA small batch can miss rare cases. | \n
| Clustering | \nGroups traces by similar content and selects examples from each group. | \nThe result depends on the features and clustering choices. | \n
| Data analysis | \nReviews extreme values such as latency or tool count. | \nAn extreme value may have nothing to do with quality. | \n
| Classification | \nUses an evaluator or another model to flag likely failures. | \nIt favors problems the classifier already knows how to find. | \n
| Feedback | \nSelects traces with negative user feedback. | \nIt misses problems that users do not report. | \n
The table above orders sampling methods from the most exploratory to the most targeted. When you’re starting out, you should optimize for exploration of the data. As you learn more, you can start to lean more heavily on signals to select traces. The proper mix of methods depends on your goals and requires experimentation.
\nKeep some random traces in every batch. This gives you a chance to find failure modes that your current signals do not describe.
\nUse targeted sampling to find rare failures. Search for signals that correlate with the failure, such as a specific tool sequence, unusually long traces, retries, or a known input pattern. Review the targeted batch to collect examples and improve the failure definition.
\nThis flashcard from our evals flashcards series visualizes these methods.
\nWe can borrow a technique from machine learning called active learning to sample production traces. In active learning, a system asks a person to label the data points that would be most useful for its next update.
\nIn Shreya Shankar’s walkthrough, Claude Code clusters traces and chooses examples from each cluster for review. A monitor command watches annotations.json for new labels. When a label arrives, the agent updates a failure taxonomy and looks for similar cases or different failures.
In the above video, active learning is used in the context of error analysis to find new cases to review. However, this approach can be used anywhere in the workflow where you are annotating data.
\n\n👉 Want to learn more about AI Evals? Check out our AI Evals course. It’s a live cohort with hands on exercises and office hours. Here is a 25% discount code for readers. 👈
\n\n\nEngineers often believe that Likert scales (1-5 ratings) provide more information than binary evaluations, allowing them to track gradual improvements. However, this added complexity often creates more problems than it solves in practice.
\n
Binary evaluations force clearer thinking and more consistent labeling. Likert scales introduce significant challenges: the difference between adjacent points (like 3 vs 4) is subjective and inconsistent across annotators, detecting statistical differences requires larger sample sizes, and annotators often default to middle values to avoid making hard decisions.
\nHaving binary options forces people to make a decision rather than hiding uncertainty in middle values. Binary decisions are also faster to make during error analysis - you don’t waste time debating whether something is a 3 or 4.
\nFor tracking gradual improvements, consider measuring specific sub-components with their own binary checks rather than using a scale. For example, instead of rating factual accuracy 1-5, you could track “4 out of 5 expected facts included” as separate binary checks. This preserves the ability to measure progress while maintaining clear, objective criteria.
\nStart with binary labels to understand what ‘bad’ looks like. Numeric labels are advanced and usually not necessary.
\n\nEach eval you create should return a binary outcome (e.g. Pass or Fail). You will likely end up with many evals, each checking a different failure. However, people in your organization may want a single number to track.
\nA simple approach I like to use is a “pass all” rate. An example passes only if it passes every check. For example, if 80 out of 100 examples pass every check, your pass-all rate is 80%. Design your report or dashboard so you can drill down from the overall pass-all rate to the pass rate for each check so you can see what’s contributing most to failures.
\nA middle ground between one overall score and a separate result for every eval is to group related checks into themes. You can then report a pass-all rate for each group. For example, reviewing Nurture Boss’s apartment leasing assistant revealed problems with conversation flow, handoffs to humans, and rescheduling. Those themes could become groups of evals.
\nAnother way to choose these groups is by how serious the failures are. For example, report one pass-all rate for checks that should block a release and another for issues you can tolerate. This approach can be helpful for gating production releases.
\nIf you still need a single score that accounts for differences in importance, you can give some checks more weight than others. I discourage complicated weighted scores for the same reason I discourage Likert scales for LLM judges. If your dashboard reports a composite score that jumps from 3.2 to 3.7 week over week, it’s easy to feel good about the increase without knowing what improved for users. In our experience, dashboards like this are usually performative and waste everyone’s time.
\nWhichever approach you choose, remember that as your eval set changes, its scores may no longer be directly comparable with older scores. Evals give you challenges to improve against, and those challenges should change as your product evolves. For tracking progress with metrics over longer time horizons, it’s often better to use product metrics in addition to evals. Measures such as churn or active users can provide a more stable basis for comparison while your evals change.
\n\nGenerally no. Eval-driven development (writing evaluators before implementing features) sounds appealing but creates more problems than it solves. Unlike traditional software where failure modes are predictable, LLMs have infinite surface area for potential failures. You can’t anticipate what will break.
\nA better approach is to start with error analysis. Write evaluators for errors you discover, not errors you imagine. This avoids getting blocked on what to evaluate and prevents wasted effort on metrics that have no impact on actual system quality.
\nException: Eval-driven development may work for specific constraints where you know exactly what success looks like. If adding “never mention competitors,” writing that evaluator early may be acceptable.
\nMost importantly, always do a cost-benefit analysis before implementing an eval. Ask whether the failure mode justifies the investment. Error analysis reveals which failures actually matter for your users.
\n\nFocus automated evaluators on failures that persist after fixing your prompts. Many teams discover their LLM doesn’t meet preferences they never actually specified - like wanting short responses, specific formatting, or step-by-step reasoning. Fix these obvious gaps first before building complex evaluation infrastructure.
\nConsider the cost hierarchy of different evaluator types. Simple assertions and reference-based checks (comparing against known correct answers) are cheap to build and maintain. LLM-as-Judge evaluators require 100+ labeled examples, ongoing weekly maintenance, and coordination between developers, PMs, and domain experts. This cost difference should shape your evaluation strategy.
\nOnly build expensive evaluators for problems you’ll iterate on repeatedly. Since LLM-as-Judge comes with significant overhead, save it for persistent generalization failures - not issues you can fix trivially. Start with cheap code-based checks where possible: regex patterns, structural validation, or execution tests. Reserve complex evaluation for subjective qualities that can’t be captured by simple rules.
\n\nFirst check whether you can test the condition with code assertions. For example, suppose an AI assistant manages your contacts, and you want to test whether it creates a contact when asked. To test this functionality, you can give it a new contact to create, then query the database to check that exactly one matching record exists with the requested details. Using code assertions avoids the need for human labels.
\nWhen a check requires judgment, use an LLM or another machine learning classifier. When using an LLM judge, we recommend using it as a classifier that returns Pass or Fail for the error you want to catch. Whichever model you use, validate it against human labels before trusting its decisions.
\nFor example, you could try Jev from TypeSafe, BERT, or logistic regression. A different model may be cheaper or faster, and it may agree more or less closely with human labels. Measure these differences on your data to find the model that meets your application’s needs. For example, you might accept slower evaluations if they catch costly failures, or prefer a faster model when you need immediate feedback.
\nWhen using an LLM, starting with a powerful model can make it easier to develop the judge’s prompt. Once it works well, try smaller, cheaper models and measure how much accuracy you lose. You can also use the same model as your application.
\nAn agent can help optimize the judge’s prompt once you have defined the task and labeled examples. Give it a specific failure to detect and a way to measure progress against your labels. “Find all errors and keep improving” is too vague. The agent needs to know what counts as an error and how to tell whether a change helped. Keep a separate test set outside the optimization process to check if the judge generalizes to examples it was not tuned against.
\n\nYes. Jev from TypeSafe is a general-purpose classifier that you can use for evals. An LLM judge that returns Pass or Fail is also a classifier.
\nYou validate Jev the same way you would any other classifier used for evals, by comparing its predictions against trusted labels. That’s why we’ve crossed out “LLM Judge” in our original flashcard and replaced it with “Classifier for Evals”:
\n![\"\"](\"https://hamel.dev/blog/posts/evals-faq/images/how-to-trust-a-classifier-for-evals.png\")
Measure against human labels and keep training, development, and test data separate to avoid overfitting.
\nTo understand the validation process described in the flashcard, see this post.
\nThe advantage of a fast inexpensive classifier (like Jev) is that it can make automated prompt tuning significantly cheaper and faster. Prompt tuning involves automatically trying changes to the evaluator’s prompt and checking whether its decisions agree more closely with human labels. GEPA is one example of a prompt tuning algorithm. Prompt tuning can sometimes require hundreds or thousands of evaluations, so a lower cost per run can add up to substantial savings.
\nNo single classifier is best for every eval. Validation with human labels help you make trade-offs between accuracy, cost, and speed for your application.
\n\nFor an evaluator that makes judgments, test it against human-labeled examples of the failure you want to detect. This applies to LLM judges and other machine learning classifiers. You need to know how often they catch failures and how often they raise false alarms. If code can directly check the condition, you do not need human labels for that check. See which model or method to use for an eval.
\nStart by splitting your labeled examples into three separate sets:
\nEach time you use dev results to change the prompt or choose a model, information from those examples influences the evaluator. After many rounds, it may do well on the dev set but poorly on new examples. This is overfitting, and it can happen even if you never put the dev examples directly in the prompt. The test set gives you a final check on data that hasn’t guided those changes.
\nIf test scores are much worse than dev scores, investigate whether you’ve overfit. Small samples make these measurements less certain, and differences between the sets can also cause a gap. If you’ve overfit, revisit the instructions and examples, then repeat development with a new, untouched test set reserved for the final check. Addressing overfitting is beyond the scope of this FAQ.
\nTo measure how well the evaluator aligns with human judgments, use the following metrics. Here, “positive” means an error is present, matching the flashcard below.
\nTrack both rates as you make changes. Catching more failures can come at the cost of more false alarms. Choose acceptable levels based on the consequences for your application. If failures are rare, even a small false-alarm rate can create a lot of unnecessary reviews.
\nThe flashcard below illustrates this process for an LLM judge. The same separation of development and testing applies to other evaluators.
\n![\"\"](\"https://hamel.dev/notes/llm/evals/flashcards/7-how-to-trust-a-llm-judge.png\")
How to trust an LLM judge: validate against human labels, separate training, development, and test examples, and measure TPR and TNR.
\nThe flashcard’s dataset split is an example for prompt-based judges or zero-shot classifiers. Training a classifier may require a larger share of training data. Choose your targets based on the cost of missed failures and false alarms in your application.
\n\nTo debug a LLM judge, you need examples with human Pass/Fail labels to compare its decisions against. An effective way to get these labels is error analysis, which provides you with a structured way to review your application’s data and find errors.
\nAs you collect labeled examples (we recommend at least 50 passing and 50 failing examples), inspect where the judge disagrees with the human labels to get clues on what needs fixing. Common issues include missing context or vague instructions. If you have trouble deciding whether an example should pass or fail, this is a sign that you need to refine your definition of success more precisely.
\nInspect a few disagreements manually before trying automated prompt tuning. Algorithms such as GEPA try changes to the judge’s prompt and measure whether they improve agreement with human labels. If you engage in prompt tuning too early, you can miss important problems that aren’t prompt related (like missing context, bad labels, etc.).
\nThe most common mistake people make is directing their LLM judge to catch too many different kinds of errors at once. Instead, we recommend building a separate judge for each type of failure. For example, checking whether the assistant escalated to a human when required is more specific than grading overall conversation quality. A focused judge is also easier to align with human labels and is more actionable.
\nFinally, make sure your judge can generalize to data you haven’t seen (i.e. its not overfitting to the data you’re tuning it with). The best way to thest this is to set aside human-labeled examples and save them for a final test. The validation FAQ explains how to split your data and measure whether the judge agrees with human reviewers on unseen examples.
\n\nNo. Generic evaluations waste time and create false confidence when you use them as quality measures. However, they can still help you find traces to inspect.
\nGeneric evaluation metrics are everywhere. Eval libraries contain scores like helpfulness, coherence, quality, etc. promising easy evaluation. These metrics measure abstract qualities that may not matter for your use case. Good scores on them don’t mean your system works.
\nInstead, conduct error analysis to understand failures. Define binary failure modes based on real problems. Create custom evaluators for those failures and validate them against human judgment.
\nExperienced practitioners may use generic metrics as exploration signals. Once you understand why they fail as quality measures, you can use them to find interesting traces for human review.
\n\nGeneric metrics like BERTScore, ROUGE, cosine similarity, etc. are not useful for evaluating LLM outputs in most AI applications. Instead, we recommend using error analysis to identify metrics specific to your application’s behavior. We recommend designing binary pass/fail.) evals (using LLM-as-judge) or code-based assertions.
\nAs an example, consider a real estate CRM assistant. Suggesting showings that aren’t available (can be tested with an assertion) or confusing client personas (can be tested with a LLM-as-judge) is problematic . Generic metrics like similarity or verbosity won’t catch this. A relevant quote from the course:
\n\n\n“The abuse of generic metrics is endemic. Many eval vendors promote off the shelf metrics, which ensnare engineers into superfluous tasks.”
\n
Similarity metrics aren’t always useless. They have utility in domains like search and recommendation (and therefore can be useful for optimizing and debugging retrieval for RAG). For example, cosine similarity between embeddings can measure semantic closeness in retrieval systems, and average pairwise similarity can assess output diversity (where lower similarity indicates higher diversity).
\n\nFor LLM-as-Judge selection, using the same model is usually fine because the judge is doing a different task than your main LLM pipeline. While research has shown that models can exhibit bias when evaluating their own outputs, what ultimately matters is how well your judge aligns with human judgments. The judges we recommend building do scoped binary classification tasks. We’ve found that iterative alignment with human labels is usually achievable on this constrained task.
\nFocus on achieving high True Positive Rate (TPR) and True Negative Rate (TNR) with your judge on a held out labeled test set. If you struggle to achieve good alignment with human scores, then consider trying a different model. However onboarding new model providers may involve non-trivial effort in some organizations, which is why we don’t advocate for using different models by default unless there’s a specific alignment issue.
\nWhen selecting judge models, start with the most capable models available to establish strong alignment with human judgments. You can optimize for cost later once you’ve established reliable evaluation criteria.
\n\nGive each judge only the parts of the trace it needs for its failure mode. Do not give every judge the same full trace by default. Extra context can cause context rot and make the judge worse.
\nFinding the right pieces of context often requires experimentation. Test your choices by comparing the judge’s decisions with human labels. Then, inspect disagreements to see whether the judge lacked necessary evidence or was distracted by irrelevant information.
\nIf you’re unsure whether a piece of information helps, try an ablation study. This means removing one piece at a time and checking how the results change against human labels. If performance stays the same or improves, you may be able to leave it out.
\nLong-running agents can produce large traces that fill or exceed the judge’s context window. For these cases, consider giving the judge a tool to search the parts it needs. However, don’t add this unless you absolutely need it, as a tool like this adds additional complexity, cost, and latency.
\n\nMany applications require a model that can refuse to answer a question when it lacks sufficient information. To evaluate whether this refusal behavior is well-calibrated, you need to test if the model refuses at the appropriate times without refusing to answer questions it should be able to answer.
\nTo do this effectively, you should construct an evaluation set that has the following components:
\nWhile the exact proportion isn’t critical, a balanced set with a roughly equal number of answerable and unanswerable questions is a good starting point. The diversity and difficulty of the questions are more important than the precise ratio.
\nThe evaluation itself is a binary (Pass/Fail) check of the model’s judgment. A “Pass” requires the model to satisfy two conditions: it must answer the answerable questions while also refusing to answer the unanswerable ones. A failure is defined as providing a fabricated answer to an unanswerable question, which indicates poor calibration.
\nIn the research literature, this capability is known as “Abstention Ability.” To improve this behavior, it is worth searching for this term on Arxiv to understand the latest techniques.
\n\nFor most small to medium-sized companies, appointing a single domain expert as a “benevolent dictator” is the most effective approach. This person becomes the definitive voice on quality standards. The expert might be a psychologist for a mental health chatbot or a lawyer for legal document analysis.
\nA single expert eliminates annotation conflicts and prevents the paralysis that comes from “too many cooks in the kitchen”. The benevolent dictator can incorporate input and feedback from others, but they drive the process. If you feel like you need five subject matter experts to judge a single interaction, it’s a sign your product scope might be too broad.
\nHowever, larger organizations or those operating across multiple domains (like a multinational company with different cultural contexts) may need multiple annotators. When you do use multiple people, you’ll need to measure their agreement using metrics like Cohen’s Kappa, which accounts for agreement beyond chance. However, use your judgment. Even in larger companies, a single expert is often enough.
\nHave annotators label the same examples independently before they discuss them. Measure agreement and collect the cases where their labels differ. During an alignment session, ask which part of the rubric caused the disagreement and what rule would make the next decision clear.
\nUpdate the rubric with a definition, rule, or example that covers the disputed case. Then relabel affected examples. If the annotators still disagree, assign a domain expert to make the final decision and record the reason.
\nStart with a benevolent dictator whenever feasible. Only add complexity when absolutely necessary.
\n\nStart by scrutinizing your product design. It’s often helpful to surface intermediate outputs users can check before a final result. For example, suppose you have an agent that writes a medical report by synthesizing a patient’s medical history. Instead of asking a doctor to provide feedback on the report, show the extracted facts with links to the source material and let doctors correct a fact or resolve conflicting evidence before generating the report. This also keeps the doctor involved and helps them build trust by checking the work as they go. This is a sketch of how such an interface might look:
\n![\"ClaimDraft](\"https://hamel.dev/blog/posts/eval-smell/_static-imgs/09-workers-comp-after.png\")
A mockup that guides a doctor through facts and conflicting evidence before generating a report.
\nFor more discussion on designing for verification, see “It’s Hard to Eval” Is a Product Smell. The post expands on this example and discusses several others with before-and-after mockups.
\nAfter you have designed for verification, make sure the review interface removes friction from reviewing data. See the advice on building a review interface. Some common tips include:
\nNext, debug the review process. First, try fewer examples so reviewers have time to inspect each one carefully. Have people review the same examples independently and discuss disagreements. You can also review examples together to see where people get stuck. Disagreement can reveal unclear instructions or missing information.
\n\nAt the outset, collaborate to establish shared context. Engineers catch technical issues like retrieval issues and tool errors. PMs identify product failures like unmet user expectations, confusing responses, or missing features users expect.
\nAs time goes on you should lean towards a benevolent dictator for error analysis: a domain expert or PM who understands user needs. Empower domain experts to evaluate actual outcomes rather than technical implementation. Ask “Has an appointment been made?” not “Did the tool call succeed?” The best way to empower the domain expert is to give them custom annotation tools that display system outcomes alongside traces. Show the confirmation, generated email, or database update that validates goal completion. Keep all context on one screen so non-technical reviewers focus on results.
\n\nYes, especially when you’re beginning with evals. I’m often surprised by the number of low-hanging fruit I find while reviewing data that don’t require domain knowledge. For example, I’ve found issues like this in specialized domains as an outsider:
\nFurthermore, ask a domain expert to walk through an example and explain why it is good or bad. Watch what they check and which evidence they need. Use what you learn to build a better annotation interface that makes reviewing easier.
\nYou can also help the team collect interactions and review them regularly. For example, see how product managers and engineers can collaborate on error analysis to get an idea of how to structure cross-functional collaboration.
\nLastly, make sure you leave judgments that require specialized knowledge to the expert. However, don’t assume you need domain expertise to start being useful!
\n\nEven though you can translate interactions in a foreign language with an LLM, be cautious about relying on it. Translation often loses meaning as some words and expressions have no direct equivalent. Moreover, what “good” means often depends on culture and social norms. For example, understanding the literal meaning of an answer is not enough to judge whether its tone is appropriate in a different cultural frame.
\nBecause of these limitations, we recommend involving a reviewer who understands both the language and the cultural context of your users. You can still contribute to evals by helping organize the review and investigating problems you can identify yourself. That reviewer should set the standard for error analysis.
\nIf you cannot find a reviewer who understands both the language and cultural context, be aware that your assessment will be limited.
\n\nOutsourcing error analysis is usually a big mistake (with some exceptions). The core of evaluation is building the product intuition that only comes from systematically analyzing your system’s failures. You should be extremely skeptical of this process being delegated.
\nWhen you outsource annotation, you often break the feedback loop between observing a failure and understanding how to improve the product. Problems with outsourcing include:
\nSuperficial Labeling: Even well-defined metrics require nuanced judgment that external teams lack. A critical misstep in error analysis is excluding domain experts from the labeling process. Outsourcing this task to those without domain expertise, like general developers or IT staff, often leads to superficial or incorrect labeling.
\nLoss of Unspoken Knowledge: A principal domain expert possesses tacit knowledge and user understanding that cannot be fully captured in a rubric. Involving these experts helps uncover their preferences and expectations, which they might not be able to fully articulate upfront.
\nAnnotation Conflicts and Misalignment: Without a shared context, external annotators can create more disagreement than they resolve. Achieving alignment is a challenge even for internal teams, which means you will spend even more time on this process.
\nInstead of outsourcing, focus on building an efficient internal evaluation process.
\n1. Appoint a “Benevolent Dictator”. For most teams, the most effective strategy is to appoint a single, internal domain expert as the final decision-maker on quality. This individual sets the standard, ensures consistency, and develops a sense of ownership.
\n2. Use a collaborative workflow for multiple annotators. If multiple annotators are necessary, follow a structured process to ensure alignment: * Draft an initial rubric with clear Pass/Fail definitions and examples. * Have each annotator label a shared set of traces independently to surface differences in interpretation. * Measure Inter-Annotator Agreement (IAA) using a chance-corrected metric like Cohen’s Kappa. * Facilitate alignment sessions to discuss disagreements and refine the rubric. * Iterate on this process until agreement is consistently high.
\nBuilding internal capacity does not mean you have to label every trace. Use these strategies to manage the workload:
\nSmart Sampling: Review a small, representative sample of traces thoroughly. It is more effective to analyze 100 diverse traces to find patterns than to superficially label thousands.
\nThe “Think-Aloud” Protocol: To make the most of limited expert time, use this technique from usability testing. Ask an expert to verbalize their thought process while reviewing a handful of traces. This method can uncover deep insights in a single one-hour session.
\nBuild Lightweight Custom Tools: Build custom annotation tools to streamline the review process, increasing throughput.
\nWhile outsourcing the core error analysis process is not recommended, there are some scenarios where external help is appropriate:
\nPurely Mechanical Tasks: For highly objective, unambiguous tasks like identifying a phone number or validating an email address, external annotators can be used after a rigorous internal process has defined the rubric.
\nTasks Without Product Context: Well-defined tasks that don’t require understanding your product’s specific requirements can be outsourced. Translation is a good example: it requires linguistic expertise but not deep product knowledge.
\nEngaging Subject Matter Experts: Hiring external SMEs to act as your internal domain experts is not outsourcing; it is bringing the necessary expertise into your evaluation process. For example, AnkiHub hired 4th-year medical students to evaluate their RAG systems for medical content rather than outsourcing to generic annotators.
\nTraces can get large when an agent runs for a long time or retrieves a large amount of context. A useful heuristic is to focus on the first upstream failure. Errors tend to compound, which means you can prioritize earlier ones to save time.
\nUse progressive disclosure in your review tool by showing the most relevant information first and letting reviewers expand details as needed. For example, show the conversation initially, with tool outputs collapsed until a reviewer needs to inspect them.
\nIf a single trace is still too large to review, work with the domain expert to identify what they need to check. Build a tool that extracts the relevant evidence and links back to its location in the trace or retrieved document. For example, when reviewing an answer about a long contract, the tool could show the relevant clauses with links to their original pages. Always validate this kind of extraction with a domain expert.
\nQuality is more important than quantity. You can usually learn more from carefully investigating a few failures than from rushing through many traces.
\n\nLLMs can speed up parts of your eval workflow, but they can’t replace human judgment where your expertise is essential. For example, if you let an LLM handle all of error analysis (i.e., reviewing and annotating traces), you might overlook failure cases that matter for your product. Suppose users keep mentioning “lag” in feedback, but the LLM lumps these under generic “performance issues” instead of creating a “latency” category. You’d miss a recurring complaint about slow response times and fail to prioritize a fix.
\nThat said, LLMs are valuable tools for accelerating certain parts of the evaluation workflow when used with oversight.
\nFirst-pass axial coding: After you’ve open coded 30–50 traces yourself, use an LLM to organize your raw failure notes into proposed groupings. This helps you quickly spot patterns, but always review and refine the clusters yourself. Note: If you aren’t familiar with axial and open coding, see this faq.
\nMapping annotations to failure modes: Once you’ve defined failure categories, you can ask an LLM to suggest which categories apply to each new trace (e.g., “Given this annotation: [open_annotation] and these failure modes: [list_of_failure_modes], which apply?”).
\nSuggesting prompt improvements: When you notice recurring problems, have the LLM propose concrete changes to your prompts. Review these suggestions before adopting any changes.
\nAnalyzing annotation data: Use LLMs or AI-powered notebooks to find patterns in your labels, such as “reports of lag increase 3x during peak usage hours” or “slow response times are mostly reported from users on mobile devices.”
\nInitial open coding: Always read through the raw traces yourself at the start. This is how you discover new types of failures, understand user pain points, and build intuition about your data. Never skip this or delegate it.
\nValidating failure taxonomies: LLM-generated groupings need your review. For example, an LLM might group both “app crashes after login” and “login takes too long” under a single “login issues” category, even though one is a stability problem and the other is a performance problem. Without your intervention, you’d miss that these issues require different fixes.
\nGround truth labeling: For any data used for testing/validating LLM-as-Judge evaluators, hand-validate each label. LLMs can make mistakes that lead to unreliable benchmarks.
\nRoot cause analysis: LLMs may point out obvious issues, but only human review will catch patterns like errors that occur in specific workflows or edge cases—such as bugs that happen only when users paste data from Excel.
\nIn conclusion, start by examining data manually to understand what’s actually going wrong. Use LLMs to scale what you’ve learned, not to avoid looking at data.
\n\nAutomating prompt engineering can be tempting, but you should be skeptical of tools that promise to optimize prompts for you, especially in early stages of development. When you write a prompt, you are forced to clarify your assumptions and externalize your requirements. Good writing is good thinking 1. If you delegate this task to an automated tool too early, you risk never fully understanding your own requirements or the model’s failure modes.
\nThis is because automated prompt optimization typically hill-climb a predefined evaluation metric. It can refine a prompt to perform better on known failures, but it cannot discover new ones. Discovering new errors requires error analysis. Furthermore, research shows that evaluation criteria tends to shift after reviewing a model’s outputs, a phenomenon known as “criteria drift” 2. This means that evaluation is an iterative, human-driven sensemaking process, not a static target that can be set once and handed off to an optimizer.
\nA pragmatic approach is to use LLMs to improve your prompt based on open coding (open-ended notes about traces). This way, you maintain a human in the loop who is looking at the data and externalizing their requirements. Once you have a high-quality set of evals, prompt optimization can be effective for that last mile of performance.
\n\n👉 Want to learn more about AI Evals? Check out our AI Evals course. It’s a live cohort with hands on exercises and office hours. Here is a 25% discount code for readers. 👈
\nBuild a custom annotation tool. This is the single most impactful investment you can make for your AI evaluation workflow. With AI-assisted development tools like Cursor or Lovable, you can build a tailored interface in hours. I often find that teams with custom annotation tools iterate ~10x faster.
\nCustom tools excel because:
\nOff-the-shelf tools may be justified when you need to coordinate dozens of distributed annotators with enterprise access controls. Even then, many teams find the configuration overhead and limitations aren’t worth it.
\nIsaac’s Anki flashcard annotation app shows the power of custom tools—handling 400+ results per query with keyboard navigation and domain-specific evaluation criteria that would be nearly impossible to configure in a generic tool.
\n\nGreat interfaces make human review fast, clear, and motivating. We recommend building your own annotation tool customized to your domain. The following features are possible enhancements we’ve seen work well, but you don’t need all of them. The screenshots shown are illustrative examples to clarify concepts. In practice, I rarely implement all these features in a single app. It’s ultimately a judgment call based on your specific needs and constraints.
\nPresent the trace in a way that’s intuitive for the domain. If you’re evaluating generated emails, render them to look like emails. If the output is code, use syntax highlighting. Allow the reviewer to see the full trace (user input, tool calls, and LLM reasoning), but keep less important details in collapsed sections that can be expanded. Here is an example of a custom annotation tool for reviewing real estate assistant emails:
\n![\"\"](\"https://hamel.dev/blog/posts/evals-faq/images/emailinterface1.webp\")
A custom interface for reviewing emails for a real estate assistant.
\nKeep reviewers in a state of flow by minimizing friction and motivating completion. Include progress indicators (e.g., “Trace 45 of 100”) to keep the review session bounded and encourage completion. Enable hotkeys for navigating between traces (e.g., N for next), applying labels, and saving notes quickly. Below is an illustration of these features:
\n![\"\"](\"https://hamel.dev/blog/posts/evals-faq/images/hotkey.webp\")
An annotation interface with a progress bar and hotkey guide
\nAllow reviewers to filter traces by metadata or search by keywords. Semantic search helps find conceptually similar problems. Clustering similar traces (like grouping by user persona) lets reviewers spot recurring issues and explore hypotheses. Below is an illustration of these features:
\n![\"\"](\"https://hamel.dev/blog/posts/evals-faq/images/group1.webp\")
Cluster view showing groups of emails, such as property-focused or client-focused examples. Reviewers can drill into a group to see individual traces.
\nSurface traces flagged by guardrails, CI failures, or automated evaluators for review. Provide buttons to take actions like adding to datasets, filing bugs, or re-running pipeline tests. Display relevant context (pipeline version, eval scores, reviewer info) directly in the interface to minimize context switching. Below is an illustration of these ideas:
\n![\"\"](\"https://hamel.dev/blog/posts/evals-faq/images/ci.webp\")
A trace view that allows you to quickly see auto-evaluator verdict, add traces to dataset or open issues. Also shows metadata like pipeline version, reviewer info, and more.
\nKeep your annotation interface minimal. Only incorporate these ideas if they provide a benefit that outweighs the additional complexity and maintenance overhead.
\n\nMost eval tools handle the basics well: logging complete traces, tracking metrics, prompt playgrounds, and annotation queues. These are table stakes. Here are four areas where you’ll likely need to supplement existing tools.
\nWatch for vendors addressing these gaps: it’s a strong signal they understand practitioner needs.
\nAfter reviewing traces where your AI fails, can your tooling automatically cluster similar issues? For instance, if multiple traces show the assistant using casual language for luxury clients, you need something that recognizes this broader “persona-tone mismatch” pattern. We recommend building capabilities that use AI to suggest groupings, rewrite your observations into clearer failure taxonomies, help find similar cases through semantic search, etc.
\nThe most effective workflows use AI to accelerate every stage of evaluation. During error analysis, you want an LLM helping categorize your open-ended observations into coherent failure modes. For example, you might annotate several traces with notes like “wrong tone for investor,” “too casual for luxury buyer,” etc. Your tooling should recognize these as the same underlying pattern and suggest a unified “persona-tone mismatch” category.
\nYou’ll also want AI assistance in proposing fixes. After identifying 20 cases where your assistant omits pet policies from property summaries, can your workflow analyze these failures and suggest specific prompt modifications? Can it draft refinements to your SQL generation instructions when it notices patterns of missing WHERE clauses?
\nGood workflows also help you conduct data analysis of your annotations and traces. I like using notebooks with AI in-the-loop like Julius or Hex. These help me discover insights like “location ambiguity errors spike 3x when users mention neighborhood names” or “tone mismatches occur 80% more often in email generation than other modalities.”
\nBe prepared to build most of your evaluators from scratch. Generic metrics like “hallucination score” or “helpfulness rating” rarely capture what actually matters for your application—like proposing unavailable showing times or omitting budget constraints from emails. In our experience, successful teams spend most of their effort on application-specific metrics.
\nCustom annotation interfaces work best for most teams. This requires observability platforms with thoughtful APIs. I often have to build my own libraries and abstractions just to make bulk data export manageable. You shouldn’t have to paginate through thousands of requests or handle timeout-prone endpoints just to get your data. Look for platforms that provide true bulk export capabilities and, crucially, APIs that let you write annotations back efficiently.
\n\nWhen building an internal eval platform, it’s tempting to start with tools, infrastructure, and a shared set of metrics. That can lead teams to adopt whatever the platform offers without checking whether it helps them find and fix problems in their products.
\nStart by encouraging teams to perform error analysis and sample data effectively for review. They can use the failures they find to decide which automated checks to build, then validate evaluators against human labels. Standardize these processes while letting each team develop its own metrics and, when needed, tools. The field guide shows an example of how these might fit together.
\nGive teams the flexibility to build their own tools, especially now that AI coding agents make custom software cheaper to create. For example, tools to annotate data often need custom interfaces that fit the data being reviewed. Reviewing text extracted from a scanned document calls for a different interface than reviewing chat conversations.
\nA platform can still provide shared storage for results and support collaboration on labeling. Start by serving one team and one use case well, then expand as you learn which needs are shared. The benefit of standardization is smaller when teams have very different needs and can build their own tools cheaply.
\nComparing eval scores across projects only makes sense when the checks and test data are comparable. We strongly advise against offering generic metrics, such as helpfulness or coherence, as a shortcut. They are rarely useful as quality measures and tend to distract teams from the failures that affect their users.
\n\nEval tools are in an intensely competitive space. It would be futile to compare their features. If I tried to do such an analysis, it would be invalidated in a week! Vendors I encounter the most organically in my work are: Langsmith, Arize and Braintrust.
\nWhen I help clients with vendor selection, the decision weighs heavily towards who can offer the best support, as opposed to purely features. This changes depending on size of client, use case, etc. Yes - it’s mainly the human factor that matters, and dare I say, vibes.
\nI have no favorite vendor. At the core, their features are very similar - and I often build custom tools on top of them to fit my needs.
\nHere is a video series that has a live commentary on the relative strengths and weaknesses of the three aforementioned vendors.
\n\nThere is an unavoidable tension between keeping prompts close to the code vs. an environment that non-technical stakeholders can access.
\nMy preferred approach is storing prompts in Git. This treats them as software artifacts that are versioned, reviewed, and deployed atomically with the application code. While the Git command line is unfriendly for non-technical folks, the GitHub web interface and the GitHub Desktop app make it very approachable. When I was working at GitHub, I worked with many non-technical professionals, including lawyers and accountants, who used these tools effectively. Here is a blog post aimed at non-technical folks to get started.
\nAlternatively, most vendors in the LLM tooling space, such as observability platforms like Arize, Braintrust, and LangSmith, offer dedicated prompt management tools. These are accessible for rapid iteration but risk creating additional layers of indirection.
\nWhy prompt management tools often fall short: AI products typically involve many moving parts: tools, RAG, agents, etc. Prompt management tools are inherently limiting because they can’t easily execute your application’s code. Even when they can, there’s often significant indirection involved, making it difficult to test prompts with your system’s capabilities.
\nWhen possible, a notebook provides a great solution for prompt experimentation If you have Python entry points into your codebase or your codebase is written in Python, Jupyter notebooks are particularly powerful for this purpose. You can experiment with prompts and iterate on your actual AI agents with their full tool and RAG capabilities. This makes it much easier to understand how your system works in practice. Additionally, you can create widgets and small user interfaces within notebooks, giving you the best of both worlds for experimentation and iteration. To see what this looks like in practice, Teresa Torres gives a fantastic, hands-on walkthrough of how she, as a PM, used notebooks for the entire eval and experimentation lifecycle:
\nIf notebooks are not feasible for your code base, an integrated prompt environment can be effective for experimentation. Either way, I prefer to version and manage prompts in Git.
\n\nNothing beats experimentation. Test both approaches (ideally with evals) with your specific model and use case. Models handle system and user prompts differently, and these differences vary by provider and model version. Move instructions between prompts and measure which produces better results for your specific task.
\nGeneral guidelines: Put static instructions and role definitions in the system prompt. Put dynamic content, examples, and task-specific details in the user prompt. Think of the system prompt as the model’s constitution—rules that apply across all requests. Include identity, behavioral constraints, output format requirements, and standing instructions: “You are a medical assistant. Never provide diagnoses. Always recommend consulting a healthcare provider.”
\nThe user prompt contains the actual task, relevant context, few-shot examples, and data to process. Documents for analysis, query-specific variations, and contextual information belong here. When the distinction feels unclear, prefer the user prompt. It’s more portable across models and easier to debug.
\n\nCI evals protect against known regressions before deployment. Online monitoring find failures in production traffic and estimate how often they occur.
\nTest datasets for CI are small (in many cases 100+ examples) and purpose-built. Examples cover core features, regression tests for past bugs, and known edge cases. Since CI tests are run frequently, the cost of each test has to be carefully considered (that’s why you carefully curate the dataset). Favor assertions or other deterministic checks over LLM-as-judge evaluators.
\nFor evaluating production traffic, you can sample live traces and run evaluators against them asynchronously. Since you usually lack reference outputs on production data, you might rely more on on more expensive reference-free evaluators like LLM-as-judge. Additionally, track confidence intervals for production metrics. If the lower bound crosses your threshold, investigate further.
\nThese two systems are complementary: when production monitoring reveals new failure patterns through error analysis and evals, add representative examples to your CI dataset. This mitigates regressions on new issues.
\nHere is a visual that helps contrast the approaches.
\n\nThere are three dimensions to consider:
\nThere are no bright-line rules. This decision often requires judgement as opposed to something formulaic. Here’s a visual that can help you think through the tradeoffs:
\n![\"How](\"https://hamel.dev/blog/posts/evals-faq/images/how-often-light.png\")
![\"How](\"https://hamel.dev/blog/posts/evals-faq/images/how-often-dark.png\")
Below are concrete examples to help you understand the factors involved. Note that these are illustrative:
\n| Eval | \nTest examples passing | \nBusiness cost of failure | \nSuggested schedule | \n
|---|---|---|---|
| An answer-quality judge with GPT-6 Astra on max reasoning. | \nAll pass | \nMedium | \nRetire or run infrequently (e.g. every 2 weeks) | \n
| A code assertion which checks that a contact was saved correctly. | \nAll pass | \nMedium | \nYou can run this on every change b/c its incredibly cheap. | \n
| A judge with Fable 5 checks whether a support agent follows a new refund policy. | \nNone pass | \nMedium | \nEven though expensive, the eval is providing useful feedback b/c nothing is passing, and the business value of catching the error is high enough. I would run this as frequently as possible. | \n
| A judge with GPT-6 Luna checks whether a support agent resolves the customer’s problem. | \nSome pass | \nMedium | \nNot a terribly expensive judge b/c model is smaller and the eval is still catching errors, so I would run this somewhat frequently (e.g. nightly). | \n
| A judge with GPT-6 Astra on max reasoning checks for improper disclosure of confidential information on a legal assistant. | \nAll pass | \nHigh | \nEven though expensive and eval is saturated, the business value of catching the error is high enough that I would run this prior to each release. Given the importance of the error, I would also try to make the eval more difficult so that it’s more useful. | \n
| A judge with GPT-6 Astra on medium reasoning checks a minor formatting preference. | \nSome pass | \nLow | \nOccasionally or retire; use code instead if possible. | \n
It’s always worth exploring cheaper evaluators to see if you can find one that provides similar or better alignment with human labels for less cost.
\nThe discussion here focused on offline evals. Online evals involve similar considerations, with an additional decision about how many production traces to sample. For example, you might run cheap checks on every trace and an expensive judge on a nightly sample. For more discussion on how these approaches work together, see How are evaluations used differently in CI/CD vs. monitoring production?
\n\nGuardrails are inline safety checks that sit directly in the request/response path. They validate inputs or outputs before anything reaches a user, so they typically are:
\nIf a guardrail triggers, the system can redact, refuse, or regenerate the response. Because these checks are user-visible when they fire, false positives are treated as production bugs; teams version guardrail rules, log every trigger, and monitor rates to keep them conservative.
\nOn the other hand, evaluators typically run after a response is produced. Evaluators measure qualities that simple rules cannot, such as factual correctness, completeness, etc. Their verdicts feed dashboards, regression tests, and model-improvement loops, but they do not block the original answer.
\nEvaluators are usually run asynchronously or in batch to afford heavier computation such as a LLM-as-a-Judge. Inline use of an LLM-as-Judge is possible only when the latency budget and reliability targets allow it. Slow LLM judges might be feasible in a cascade that runs on the minority of borderline cases.
\nApply guardrails for immediate protection against objective failures requiring intervention. Use evaluators for monitoring and improving subjective or nuanced criteria. Together, they create layered protection.
\nWord of caution: Do not use llm guardrails off the shelf blindly. Always look at the prompt.
\n\nYes, but only a specific subset of them. This is the distinction between an evaluator and a guardrail that we previously discussed. As a reminder:
\nEvaluators typically run asynchronously after a response has been generated. They measure quality but don’t interfere with the user’s immediate experience.
\nGuardrails run synchronously in the critical path of the request, before the output is shown to the user. Their job is to prevent high-impact failures in real-time.
\nThere are two important decision criteria for deciding whether to use an evaluator as a guardrail:
\nLatency & Cost: Can the evaluator run fast enough and cheaply enough in the critical request path without degrading user experience?
\nError Rate Trade-offs: What’s the cost-benefit balance between false positives (blocking good outputs and frustrating users) versus false negatives (letting bad outputs reach users and causing harm)? In high-stakes domains like medical advice, false negatives may be more costly than false positives. In creative applications, false positives that block legitimate creativity may be more harmful than occasional quality issues.
\nMost guardrails are designed to be fast (to avoid harming user experience) and have a very low false positive rate (to avoid blocking valid responses). For this reason, you would almost never use a slow or non-deterministic LLM-as-Judge as a synchronous guardrail. However, these tradeoffs might be different for your use case.
\n\nMany developers fixate on model selection as the primary way to improve their LLM applications. Start with error analysis to understand your failure modes before considering model switching. As Hamel noted in office hours, “I suggest not thinking of switching model as the main axes of how to improve your system off the bat without evidence. Does error analysis suggest that your model is the problem?”
\n\nQuestion: Should I avoid using RAG for my AI application after reading that “RAG is dead” for coding agents?
\n\n\nMany developers are confused about when and how to use RAG after reading articles claiming “RAG is dead.” Understanding what RAG actually means versus the narrow marketing definitions will help you make better architectural decisions for your AI applications.
\n
The viral article claiming RAG is dead specifically argues against using naive vector database retrieval for autonomous coding agents, not RAG as a whole. This is a crucial distinction that many developers miss due to misleading marketing.
\nRAG simply means Retrieval-Augmented Generation - using retrieval to provide relevant context that improves your model’s output. The core principle remains essential: your LLM needs the right context to generate accurate answers. The question isn’t whether to use retrieval, but how to retrieve effectively.
\nFor coding applications, naive vector similarity search often fails because code relationships are complex and contextual. Instead of abandoning retrieval entirely, modern coding assistants like Claude Code still uses retrieval —they just employ agentic search instead of relying solely on vector databases, similar to how human developers work.
\nYou have multiple retrieval strategies available, ranging from simple keyword matching to embedding similarity to LLM-powered relevance filtering. The optimal approach depends on your specific use case, data characteristics, and performance requirements. Many production systems combine multiple strategies or use multi-hop retrieval guided by LLM agents.
\nUnfortunately, “RAG” has become a buzzword with no shared definition. Some people use it to mean any retrieval system, others restrict it to vector databases. Focus on the ultimate goal: getting your LLM the context it needs to succeed. Whether that’s through vector search, agentic exploration, or hybrid approaches is a product and engineering decision.
\nRather than following categorical advice to avoid or embrace RAG, experiment with different retrieval approaches and measure what works best for your application. For more info on RAG evaluation and optimization, see this series of posts.
\n\nIf your coding agent handles a wide variety of tasks, start by using public benchmarks much as you would a foundation model. For an agent that handles a narrow workflow, product-specific evals are a better fit. The evals FAQ explains this distinction.
\nPopular coding benchmarks include SWE-bench, Terminal-Bench, Aider Polyglot, and HumanEval.
\nIn addition to public benchmarks, you can also build a private benchmark of difficult tasks from your organization. OpenAI described using real internal software engineering tasks to evaluate Codex at launch. Each task needs a working environment and code-based tests that establish whether the agent completed it successfully.
\nTo decide which tasks to include, look at how people use your agent and where it fails. Review runs with engineers, group recurring problems, and turn useful examples into tests. This is error analysis, and it applies to coding products too. If existing tests already identify failures, use those results to choose runs to investigate.
\nAnthropic’s Clio research illustrates a related approach that clusters chat conversations by topic. You can apply that idea to coding sessions to identify the kinds of work your benchmark should cover.
\nAnthropic’s coding-agent eval guidance recommends starting with clearly specified tasks and a stable environment where unit tests can verify results. After you have these unit tests, they recommend adding checks for things those tests don’t capture, such as code quality or how the agent interacts with users. Claude Code’s team, for example, added evals for file edits and later for over-engineering. There are many approaches to measure file edits and over-engineering but you can start with metrics like net new lines of code added and cyclomatic complexity.
\nJohn Berryman and Shawn Simister’s Copilot talk provides additional examples of coding-agent evals. For code completions, the team removed function implementations from repositories, had the model regenerate them, and ran the existing tests. For chat, they used LLM judges with specific criteria and separate checks for whether the assistant called the right tool. They also ran A/B tests, tracking whether users accepted suggestions and kept the code afterward. These product metrics complemented the offline evals.
\n\nRAG systems have two distinct components that require different evaluation approaches: retrieval and generation.
\nThe retrieval component is a search problem. Evaluate it using traditional information retrieval (IR) metrics. Common examples include Recall@k (of all relevant documents, how many did you retrieve in the top k?), Precision@k (of the k documents retrieved, how many were relevant?), or MRR (how high up was the first relevant document?). The specific metrics you choose depend on your use case. These metrics are pure search metrics that measure whether you’re finding the right documents (more on this below).
\nTo evaluate retrieval, create a dataset of queries paired with their relevant documents. Generate this synthetically by taking documents from your corpus, extracting key facts, then generating questions those facts would answer. This reverse process gives you query-document pairs for measuring retrieval performance without manual annotation.
\nFor the generation component, check how well the LLM uses the retrieved context and whether it answers the question. Use error analysis to identify failure modes, collect human labels, build targeted LLM judges, and validate those judges against human annotations.
\nJason Liu’s “There Are Only 6 RAG Evals” provides a framework that maps well to this separation. His Tier 1 covers traditional IR metrics for retrieval. Tiers 2 and 3 evaluate relationships between Question, Context, and Answer. These include whether the context is relevant (C|Q), whether the answer is faithful to context (A|C), and whether the answer addresses the question (A|Q).
\nIn addition to Jason’s six evals, error analysis on your specific data may reveal domain-specific failure modes that warrant their own metrics. For example, a medical RAG system might consistently fail to distinguish between drug dosages for adults versus children, or a legal RAG might confuse jurisdictional boundaries. These patterns emerge only through systematic review of actual failures. Once identified, you can create targeted evaluators for these specific issues beyond the general framework.
\nFinally, when implementing Jason’s Tier 2 and 3 metrics, don’t just use prompts off the shelf. The standard LLM-as-judge process requires several steps: error analysis, prompt iteration, creating labeled examples, and measuring your judge’s accuracy against human labels. Once you know your judge’s True Positive and True Negative rates, you can correct its estimates to determine the actual failure rate in your system. Skip this validation and your judges may not reflect your actual quality criteria.
\nIn summary, debug retrieval first using IR metrics, then tackle generation quality using properly validated LLM judges.
\n\nUnlike RAG, where chunks are optimized for retrieval, document processing assumes the model will see every chunk. The goal is to split text so the model can reason effectively without being overwhelmed. Even if a document fits within the context window, it might be better to break it up. Long inputs can degrade performance due to attention bottlenecks, especially in the middle of the context. Two task types require different strategies:
\nThese are tasks where the output length doesn’t grow with input: extracting a number, answering a specific question, classifying a section. For example:
\nUse the largest chunk (with caveats) that likely contains the answer. This reduces the number of queries and avoids context fragmentation. However, avoid adding irrelevant text. Models are sensitive to distraction, especially with large inputs. The middle parts of a long input might be under-attended. Furthermore, if cost and latency are a bottleneck, you should consider preprocessing or filtering the document (via keyword search or a lightweight retriever) to isolate relevant sections before feeding a huge chunk.
\nThese include summarization, exhaustive extraction, or any task where output grows with input. For example:
\nIn these cases, smaller chunks help preserve reasoning quality and output completeness. The standard approach is to process each chunk independently, then aggregate results (e.g., map-reduce). When sizing your chunks, try to respect content boundaries like paragraphs, sections, or chapters. Chunking also helps mitigate output limits. By breaking the task into pieces, each piece’s output can stay within limits.
\nIt’s important to recognize why chunk size affects results. A larger chunk means the model has to reason over more information in one go – essentially, a heavier cognitive load. LLMs have limited capacity to retain and correlate details across a long text. If too much is packed in, the model might prioritize certain parts (commonly the beginning or end) and overlook or “forget” details in the middle. This can lead to overly coarse summaries or missed facts. In contrast, a smaller chunk bounds the problem: the model can pay full attention to that section. You are trading off global context for local focus.
\nNo rule of thumb can perfectly determine the best chunk size for your use case – you should validate with experiments. The optimal chunk size can vary by domain and model. I treat chunk size as a hyperparameter to tune.
\n\nStart simple. Check if the whole conversation met the user’s goal with a pass/fail judgment. Look at the entire trace and focus on the first upstream failure. Read the user-visible parts first to understand if something went wrong. Only then dig into the technical details like tool calls and intermediate steps.
\nFor multi-agent flows, assign a session or trace ID to each user request and log every message with its source (which agent or tool), trace ID, and position in the sequence. This lets you reconstruct the full path from initial query to final result across all agents.
\nAnnotate only the first failure in the trace at first. Downstream failures often cascade from the first issue, so fixing the upstream failure can resolve the dependent ones. As you gain experience, you can annotate independent failure modes within the same trace to speed up error analysis.
\nWhen you find a failure, reproduce it with the simplest possible test case. Here’s an example: suppose a shopping bot gives the wrong return policy on turn 4 of a conversation. Before diving into the full multi-turn complexity, simplify it to a single turn: “What is the return window for product X1000?” If it still fails, you’ve proven the error isn’t about conversation context - it’s likely a basic retrieval or knowledge issue you can debug more easily.
\nYou have two main approaches. First, simulate users with another LLM to create realistic multi-turn conversations. Second, use “N-1 testing” where you provide the first N-1 turns of a real conversation and test what happens next. The N-1 approach often works better since it uses actual conversation prefixes rather than fully synthetic interactions, but is less flexible.
\nThe key is balancing thoroughness with efficiency. Not every multi-turn failure requires multi-turn analysis.
\nWhen the conversation includes tools or several agents, use a transition failure matrix to find hotspots of errors.
\n\nCapture the complete user journey in your traces, including human handoffs. The trace continues until the user’s need is resolved or the session ends, not when AI hands off to a human. Log the handoff decision, why it occurred, context transferred, wait time, human actions, final resolution, and whether the human had sufficient context. Many failures occur at handoff boundaries where AI hands off too early, too late, or without proper context.
\nEvaluate handoffs as potential failure modes during error analysis. Ask: Was the handoff necessary? Did the AI provide adequate context? Track both handoff quality and handoff rate. Sometimes the best improvement reduces handoffs entirely rather than improving handoff execution.
\n\nLog the entire workflow from initial trigger to final business outcome. Include LLM calls, tool usage, human approvals, and database writes in your traces. You will need this visibility to properly diagnose failures.
\nUse both outcome and process metrics. Outcome metrics verify the final result meets requirements: Was the business case complete? Accurate? Properly formatted? Process metrics evaluate efficiency: step count, time taken, resource usage. Process failures are often easier to debug since they’re more deterministic, so tackle them first.
\nSegment your error analysis by workflow stages. Early stage failures (understanding user input) differ from middle stage failures (data processing) and late stage failures (formatting output). Early stage improvements have more impact since errors cascade in LLM chains.
\nUse transition failure matrices to analyze where workflows break. Create a matrix showing the last successful state versus where the first failure occurred. This reveals failure hotspots and guides where to invest debugging effort.
\n\nWe recommend evaluating agentic workflows in two phases:
\n1. End-to-end task success. Treat the agent as a black box and decide whether it met the user’s goal. Define a precise success rule per task and measure it with human review or validated LLM judges. Record the first upstream failure during error analysis.
\nOnce error analysis reveals which workflows fail most often, move to step-level diagnostics to understand why they’re failing.
\n2. Step-level diagnostics. After you log the system’s traces, you can score individual components such as:
\nTest the tool name, arguments, result, and resulting state as separate checks. Use code assertions when the expected behavior is objective. For example, verify that the agent selected cancel_order, passed the correct order ID, received a successful response, and changed the order status before it told the user that cancellation succeeded.
Also test authorization and preconditions. A valid tool call can still be wrong if the user did not approve the action or the system skipped a required check.
\nExample: “Find Berkeley homes under $1M and schedule viewings” breaks into: parameters extracted correctly, relevant listings retrieved, availability checked, and calendar invites sent. Each checkpoint can pass or fail independently, making debugging tractable.
\nUse transition failure matrices to understand error patterns. Create a matrix where rows represent the last successful state and columns represent where the first failure occurred. This is a great way to understand where the most failures occur.
\n![\"\"](\"https://hamel.dev/blog/posts/evals-faq/images/shreya_matrix.webp\")
Transition failure matrix showing hotspots in text-to-SQL agent workflow
\nTransition matrices show where failures cluster. In this example, GenSQL → ExecSQL transitions cause 12 failures while DecideTool → PlanCal causes only 2. The counts show where to investigate first. Here is another text-to-SQL example from Bryan Bischof:
\n![\"\"](\"https://hamel.dev/blog/posts/evals-faq/images/bischof_matrix.webp\")
Bischof, Bryan “Failure is A Funnel - Data Council, 2025”
\nIn this example, Bryan shows variation in transition matrices across experiments. How you organize your transition matrix depends on the specifics of your application. For example, Bryan’s text-to-SQL agent has an inherent sequential workflow which he exploits for further analytical insight. You can watch his full talk for more details.
\nCreating Test Cases for Agent Failures
\nCreating test cases for agent failures follows the same principles as our previous FAQ on debugging multi-turn conversation traces. Reproduce the error with the simplest test that still fails. Use a multi-turn test only when the failure depends on conversation context.
\n\n👉 Want to learn more about AI Evals? Check out our AI Evals course. It’s a live cohort with hands on exercises and office hours. Here is a 25% discount code for readers. 👈
\nPaul Graham, “Writes and Write-Nots” ↩
\nShreya Shankar, et al., “Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences” ↩
\nFor the past 3 years, AI evals have been my professional focus.1 The most common objection I hear to evals is “our product is hard to eval”.
\nThis objection is a product smell. Artifacts that are hard for you to verify are often hard for users too. In the worst case, users have to redo the work from scratch to verify the output. More importantly, designing your product for ease of verification should come before building evals.
\nIn this post, I’ll walk through three products I advised on that faced this issue. I’ll also show before and after sketches to demonstrate design principles. After these examples, I’ll discuss how to apply this general pattern to your product.
\nAlmost every company I’ve worked with builds an internal AI data agent. You ask it a business question, like what was net revenue for Product A last quarter, and it finds relevant data sources, runs the queries, and provides an answer. The goal of this agent is to reduce dependency on data analysts.
\nA common mistake when building AI data agents is to make the answer the only output, as illustrated below.
\nData Agent
\nWhat was net revenue for Product A last quarter?
\nNet revenue for Product A last quarter was $4.21M.
\nAsk anything about your business…➤
\nSince the only output is the answer, there is nothing here to check.
\nIn the sketch above, the user has no way to verify the answer beyond redoing work.2 A better design is to provide the user with checkable artifacts, informed by how a domain expert might validate the output. Here are techniques I use to validate metrics as a data scientist:
\nHere’s what a better interface might look like. The two tabs below show the same answer at two levels of detail. The chat reply surfaces the details worth seeing up front, and the notebook holds the full analysis behind the answer. Use the tabs to switch between them.
\n\nData Agent
\nWhat was net revenue for Product A last quarter?
\nNet revenue for Product A last quarter was $4.21M.ⓘ
\nThe agent surfaces the important details behind the answer. Select the Notebook tab above to see the full analysis generated by the agent.
\nThis is the notebook the agent worked in while producing its answer. Scroll within the figure to see all the cells. The sidebar has a Contents tab for jumping between sections and an Assistant tab for asking follow-ups.
\nnet_revenue_product_a.ipynb ▶ Run all ↑ Publish
\nMarkdown B i H 🔗 ☰
\n“What was net revenue for Product A last quarter?”
\nShort answer: $4.21M. This notebook shows how that number was built and what was checked against a trusted source.
\nNet revenue is gross − returns − discounts. Read the definition from the governed metrics layer so this matches what finance reports.
[1]
\nPython ▶ Code collapsed ✦ Ask AI ⌘K
\nimport yaml\n\ndefn = yaml.safe_load(open("metrics/net_revenue.yml"))\ndefn["expr"], defn["source"]\n( 'gross - returns - discounts', 'finance.order_lines')
\nPull net revenue for Product A for the quarter, straight from the order lines.
\n[2]
\nSQL ▶ Code collapsed ✦ Ask AI ⌘K
\nSELECT SUM(gross - returns - discounts) AS net_revenue\nFROM finance.order_lines\nWHERE product = 'Product A'\n AND fiscal_quarter = 'Q4-FY25';\n| net_revenue | \n
|---|
| 4,210,442 | \n
That's the $4.21M the chat answer reported.
\nBreak Product A down by region, then plot it. Nothing should look out of place against last quarter's mix.
\n[3]
\nSQL ▶ Code collapsed ✦ Ask AI ⌘K
\nSELECT region,\n SUM(gross - returns - discounts) AS net_revenue\nFROM finance.order_lines\nWHERE product = 'Product A'\n AND fiscal_quarter = 'Q4-FY25'\nGROUP BY region;\n| region | \nnet_revenue | \n
|---|---|
| West | \n1,740,000 | \n
| Central | \n1,160,000 | \n
| East | \n890,000 | \n
| Intl | \n420,000 | \n
[4]
\nPython ▶ Code collapsed ✦ Ask AI ⌘K
\nm = by_region.set_index("region")["net_revenue"] / 1e6\nm.plot.barh(title="Net revenue by region · Q4 FY25")\nNet revenue by region · Q4 FY25
\nWest 1.74
\nCentral 1.16
\nEast 0.89
\nIntl 0.42
\n0 0.5 1.0 1.5 2.0
\nnet revenue ($M)
\nWest and Central drive most of the revenue, with International a small tail.
\nTwo inputs have no trusted source to check against. Instead of treating them as settled, each is left below as a cell you can run and edit to dig in.
\n[5]
\nSQL ▶ Code collapsed ✦ Ask AI ⌘K
\n-- the agent's -$0.7M returns figure is an estimate;\n-- check the returns table for Q4 rows to back it\nSELECT COUNT(*) AS n_rows, SUM(amount) AS returns\nFROM finance.returns\nWHERE product = 'Product A'\n AND fiscal_quarter = 'Q4-FY25';\n| n_rows | \nreturns | \n
|---|---|
| 0 | \nNULL | \n
# Q4 returns haven't loaded yet, so the −$0.7M is an estimate
\nSo that −$0.7M has no source to check against yet. The other open item is the customer join:
\n[6]
\nSQL ▶ Code collapsed ✦ Ask AI ⌘K
\n-- 183 of 12,480 unique customers are in Billing, not CRM,\n-- so some revenue can't be attributed. Pull them:\nSELECT b.customer_id, b.amount\nFROM billing.invoices b\nLEFT JOIN crm.customers c USING (customer_id)\nWHERE c.customer_id IS NULL\nORDER BY b.amount DESC\nLIMIT 5;\n| customer_id | \namount | \n
|---|---|
| BIL-44821 | \n18,400 | \n
| BIL-39105 | \n12,950 | \n
| BIL-50277 | \n9,310 | \n
| … | \n… | \n
# 183 rows · $61,540 unattributed
\nThat's $61,540 of revenue whose customer attribution is uncertain, surfaced so a person can resolve it before the total is trusted.
\nContents ✦Assistant
\nNew thread ✕
\nWhat was net revenue for Product A last quarter?
\n✦
\nI pulled Product A's net revenue from the order lines using finance's governed definition, broke it out by region, and flagged what I couldn't verify. The cells are on the left.
\n✓ Generated the cells in this notebook
\nAsk a follow-up…
\nSQL [2] Auto ▾ ↑
\nThe notebook reads top to bottom: the assumptions the agent made, the queries it ran, and an explicit list of what it could not verify.
\nThere is a lot to unpack here. Here are notable changes:
\nThis design sketch is far from perfect. The point is that the product should help the user verify the answer as a domain expert would. Compare it to the earlier approach, where the only output was the number.
\nData agents like these are not science fiction. Hex4 is my favorite product in this genre; it integrates notebooks and chat better than anything I’ve seen. Here are screenshots from their landing page:
\n![\"\"](\"https://hamel.dev/blog/posts/eval-smell/hex-chat.png\")
Chat interface.
\n![\"\"](\"https://hamel.dev/blog/posts/eval-smell/hex-notebook.png\")
Notebook view which allows the user to see the intermediate steps and data.
\nBut what does this have to do with evals? If you design your product for verification, annotation becomes less expensive and evals will have better signals to draw from. More importantly, you’ll provide your users with a better product.
\nA founder I advised was building an AI tool that writes physical education lesson plans for K-12 teachers. A teacher enters their constraints, like the grade they teach, how long the class is, whether it meets indoors or outdoors, and what equipment they have. The tool then writes a lesson plan for those constraints. The goal is to save teachers the time they spend planning and give them a plan that fits their class. Here is a sketch of what the product looks like:
\nPE Planner
\nClass details
\nGrade Grade 4▾
\nClass length 45 minutes▾
\nLocation Outdoors▾
\nEquipment 12 cones, 6 balls
\nGenerate lesson plan
\nThe teacher enters constraints and the tool writes a plan from scratch. The only output is the plan, so its difficult to verify.
\nThe founder asked me how to eval the lesson plans. I turned the question around: what does a teacher care about?
\nThe fastest way to trust a plan is to see that a teacher like them already uses it. Additionally, teachers value visibility into what others are doing so they can learn new approaches. Therefore, a better design might start from vetted lesson plans that are actively used in schools. When the tool generates a plan, it shows which vetted plan it started from, who uses that plan, and a diff of what it changed for this teacher’s constraints.
\nNext, the teacher can check a small set of changes against a plan they already trust, instead of judging a whole plan from scratch. Here’s a sketch of what a better interface might look like:
\nPE Planner
\n\nDR Adapted from Dana Ruiz's plan
\nGrade 4 PE · Lincoln Elementary, Austin TX
\nUsed at 14 schools Run 30+ times this year Aligned to SHAPE standards
\nFull plan
\nWhat changed · 2 edits Accept all Reject all
\n1 Warm-up
\n− Jog 2 laps, then dynamic stretches (10 min)
\nShortened to fit a 45-minute class
\nAccept ⌘⏎ Reject ⌘⌫
\n2 Main activity
\n− 4 stations, 8 balls, hoops as targets
\nMatched to your equipment
\nAccept ⌘⏎ Reject ⌘⌫
\nThe plan is anchored to a vetted plan another teacher uses. The changes are marked so the teacher can check a diff against a plan they trust.
\nIn this version, most of the plan is inherited from a vetted plan. The teacher’s review is scoped to a few edits, each with a reason explaining why the change was made. This is a more efficient way to review a plan because it reduces the cognitive load of judging a whole plan from scratch.
\nDesigning for this makes the product simpler to build. Instead of stuffing hundreds of examples into a prompt, the tool captures important dimensions, retrieves a close match, and adapts it. Automated evals now become tractable because there is less surface area to test. For example, you can verify that the plan retrieval picked a sensible anchor, and each edit honors the constraints.
\nThe last example comes from a workers’ compensation tool a founder asked me to help with. It reads a patient’s chart (intake forms, imaging reports, therapy notes, prior exams) and generates a long expert opinion report, often fifty pages or more. Here is a sketch of the product:
\nClaimDraft – □ ✕
\nSource records · 18
\nIntake questionnaire
\nMRI, lumbar spine
\nX-ray report
\nPhysical therapy notes
\nTreating physician notes
\nPrior IME
\nWork restrictions
\n+ 11 more
\nClaimant J. Doe Claim #WC-20259417 Date of injury Mar 3, 2025 Examiner Dr. A. Patel
\nIndependent Medical Evaluation
\nPermanent & Stationary Report
\n1. History of injury
\nThe claimant is a 47-year-old warehouse associate who reports a lumbar spine injury on March 3, 2025 while lifting a carton estimated at sixty pounds. He describes immediate low back pain radiating into the right lower extremity, followed by numbness along the lateral calf.
\n2. Review of records
\nRecords reviewed include the intake questionnaire, an MRI of the lumbar spine dated March 18, 2025, twelve physical therapy notes, and the treating physician's progress reports through August 2025.
\n3. Physical examination
\nOn examination, lumbar flexion was limited to 40 degrees with pain. Straight leg raise was positive on the right at 50 degrees. Strength was 4 of 5 in the right extensor hallucis longus, with diminished sensation in the L5 distribution.
\n4. Diagnoses
\nLumbar disc herniation at L5-S1 with associated right L5 radiculopathy, supported by the imaging and examination findings above.
\n5. Causation analysis
\nWithin reasonable medical probability, the disc herniation is causally related to the industrial lifting event of March 3, 2025. The claimant had no documented history of lumbar treatment prior to that date.
\nPage 1 of 52 Generated from 18 records
\nThe only output is a fifty-page narrative. To trust it, the doctor has to re-read the whole chart and check every claim, which can take as long as writing the report from scratch.
\nThe problem is the same as the other examples, but the stakes are higher. The only output is the report, and the doctor is the one accountable for it. To trust it they have to go back through the chart and confirm the facts and inferences themselves. That can take as long as writing the report from scratch, which defeats the point of the tool.
\nYou might object that a fifty-page opinion is hard to verify. That is true, and the product should not pretend otherwise. Helping a doctor understand the evidence is arguably more valuable than the finished document. Therefore, I advised the founder to make the product work like a research assistant instead of a report generator.
\nFor example, the product could read every record and pull out relevant facts, with a link back to the page so the doctor can check each one. Where two exams disagree, or the chart leaves a question open, the product should surface that. The doctor can then resolve any contradictions and fill in the gaps. Finally, the product can assemble the final report from what they have already checked. Here is a sketch of what this might look like:
\nClaimDraft – □ ✕
\nClaimant J. Doe Claim #WC-20259417 Date of injury Mar 3, 2025 Examiner Dr. A. Patel
\nRecords · 18
\nMRI, lumbar spine✓
\nPhysical therapy notes✓
\nTreating physician notes✓
\nPrior IME○
\nIntake questionnaire○
\nX-ray report○
\n9 of 18 reviewed
\n1 Contradiction
\nStraight-leg-raise is recorded as positive on the right by the treating physician (p. 31) and negative by the prior IME (p. 22).
\n2 Key fact
\nThe intake form notes a prior lumbar strain in 2019, which bears on causation (p. 14).
\nInclude Dismiss
\n3 Open question
\nNo imaging appears in the file after March 18, 2025, so the current status of the herniation is unconfirmed.
\nAdd a note
\n4 Key fact
\nThe MRI report describes pre-existing degenerative changes at L4-L5 (p. 9).
\nInclude Dismiss
\n6 of 24 findings still need review Generate report draft
\nThe product walks the doctor through the evidence first, surfacing each finding with a link back to the chart. Afterwards, the final report is built from the facts they reviewed.
\nThe research assistant version of this product allows the doctor to build trust by verifying facts as they go. Similar to the other examples, this design is easier to build and evaluate. Now there are scoped units to grade, such as whether a contradiction is real or whether a citation supports a claim.
\nIt is important to understand how users verify your product’s AI artifacts. Sometimes, this may require assembling supporting evidence. In other cases, it could mean refactoring the entire workflow so that the user is in the loop (like the workers’ comp example).
\nBelow are questions that can guide your product’s design for verification:
\nA common thread across these examples is provenance. The fastest way to make an output checkable is to show where each part came from, with links to see more detail. Additionally, you can use progressive disclosure so these sources don’t overwhelm the user.
\nWhat needs verifying also changes as the user’s trust grows. Early on, the data agent should make provenance obvious, like where a metric definition came from. Once the user trusts the agent gets it right, that detail can collapse by default in the card. Good design meets users where they are instead of showing everything.
\nThese principles hold even when a product seems easy to eval. Coding is a good example: it is one of the most verifiable kinds of work, with tests, types, and diffs. Even so, some coding agents go the extra mile to make their work checkable. Cursor and Devin both record a short video of the UI changes they make, so you can confirm the work is right without reproducing it yourself.5
\nEvals thinking is aligned with good product design. Gathering supporting data and breaking down workflows into smaller units makes automated grading easier. However, I don’t want to pretend like any wisdom here is new.
\nAll of these ideas stem from well-established design principles. For example, watching an expert work to learn what they check before you build is called needfinding.6 In research-heavy work like the medical case, there is a design goal called sensemaking, which is the work of building a structured understanding of a body of evidence you can reason over.7 There are many other concepts, but I think you get the idea.
\nEven though these ideas are well established, a reminder is due in the age of AI. Before AI, verification often happened incidentally during the process of creating work product. With AI, verification is the bottleneck. It is time to think about it more explicitly.
\nTo stay updated on my writing, subscribe to my newsletter below.
\nI recorded a short video walking through these examples. If you’d rather watch than read, here it is:
\n\nThanks to Shreya Shankar and Isaac Flath for feedback on this post.
\nThe AI evals topic hub collects the rest of my evals work. I also co-teach the AI Evals for Engineers and PMs course. ↩
\nLenny Rachitsky tweeted about this recently: most of one data science team’s work is now reviewing half-baked AI analysis from PMs and engineers, and half of it is wrong. ↩
\nWhen I worked at Airbnb we had an internal tool called the Knowledge Repo, a place where data scientists published notebooks of their deep dives on analytics, modeling, and so on (they wrote about it here). It was one of the fastest ways to get context on a new project, since you could read what someone had already worked out. I don’t know if it is still in use, but the paradigm is a good one. ↩
\nBryan Bischof led the creation of the AI at Hex. Bryan is a data scientist himself, the kind of domain expert the product serves, and I think that is part of why it is designed well. ↩
\nSee a demo of Cursor recording its UI changes, and Devin’s documentation on testing and recordings. ↩
\nDev Patnaik and Robert Becker, “Needfinding: The Why and How of Uncovering People’s Needs”, Design Management Journal. ↩
\nDaniel M. Russell, Mark J. Stefik, Peter Pirolli, and Stuart K. Card, “The Cost Structure of Sensemaking”. ↩
\nIs the heyday of the data scientist over? The Harvard Business Review once called it “The Sexiest Job of the 21st Century.”1 In tech, data scientist roles were often among the best paid.2 The job also demanded an unusual mix of skills:
\n\n\nData Scientist (n.): Person who is better at statistics than any software engineer and better at software engineering than any statistician.
\n— JosH100 (@josh_wills) May 3, 2012
\n
In addition to creating a high-barrier to entry, these skills enabled data scientists to build predicitive models, measure casuality and find patterns in data. Of these, predicitive modeling paid best. Companies later peeled that work off into a new title: Machine Learning Engineer (“MLE”).3
\nFor years, shipping AI meant keeping data scientists and MLEs on the critical path. With LLMs, this stopped being the default. Foundation-model APIs now allow teams to integrate AI independently.
\nGetting cut out of the loop rattled data scientists and MLEs I know. If the company no longer needs you to ship AI, it is fair to wonder whether the job still has the same upside. The harsher story people tell themselves: unless you are pretraining at a foundation-model lab, you are not where the action is.
\nIn my opinion, training models was never most of the job. The bulk of the work is setting up experiments to test how well the AI generalizes to unseen data, debugging stochastic systems, and designing good metrics. Calling an LLM over an API does not make this work go away.
\nI recently gave a talk titled “The Revenge of the Data Scientist” at PyAI Conf to make that case with examples rather than assertion alone. Below is an annotated version of that presentation.
\n![\"\"](\"https://hamel.dev/blog/posts/revenge/images/slide_1.png\")
![\"\"](\"https://hamel.dev/blog/posts/revenge/images/slide_2.png\")
OpenAI published a blog post on harness engineering that I recommend reading. They describe how Codex worked on a software project for months, autonomously, with agents developing code bounded by a harness of tests and specifications.
\n![\"\"](\"https://hamel.dev/blog/posts/revenge/images/slide_3.png\")
One detail in that blog post’s description of the harness is easy to miss. It includes an observability stack: logs, metrics, and traces exposed to the agent so it can tell when it is going off track. In addition to tests and specifications, there are metrics. That is a key component of the system.
\n![\"\"](\"https://hamel.dev/blog/posts/revenge/images/slide_4.png\")
Andrej Karpathy’s auto-research project shows the same pattern: models iteratively optimize against a validation loss metric. Same idea, different harness.
\n![\"\"](\"https://hamel.dev/blog/posts/revenge/images/slide_5.png\")
What I want to convince you of is that a large portion of the harness is data science.
\nLet’s take a step back and take stock of where we are.
\n![\"\"](\"https://hamel.dev/blog/posts/revenge/images/slide_6.png\")
Years ago, practitioners spent hours examining data, checking label alignment, and designing metrics. Today, we build on “vibes,” ask the model if it did a good job, and grab off-the-shelf metric libraries without looking at the data.
\n![\"\"](\"https://hamel.dev/blog/posts/revenge/images/slide_7.png\")
This shows up most around retrieval and evals. Without a data background, engineers fear what they don’t understand. They claim “RAG is dead” or “evals are dead,” yet build systems that depend on those concepts.
\nThe rest of this post walks through five eval pitfalls I see repeatedly, and what a data scientist would do differently in each case.
\nThe first pitfall is generic metrics.
\n![\"\"](\"https://hamel.dev/blog/posts/revenge/images/slide_9.png\")
It is tempting to reach for an eval framework and use its metrics off the shelf. The problem: you have no idea what is actually broken. Most teams put up a dashboard with helpfulness scores, coherence scores, hallucination scores. These sound reasonable. They are also generic enough to be useless for diagnosing your application’s failures.
\nA data scientist would not adopt metrics off the shelf. They would explore the data, explore the traces, ask “what is actually breaking here?”, and figure out the highest-value thing to start measuring. There are infinite things to measure. You have to form hypotheses and iterate.
\nThe best medicine for this pitfall is looking at the data.
\n![\"\"](\"https://hamel.dev/blog/posts/revenge/images/slide_12.png\")
What does “looking at the data” mean in practice? It means reading traces. Code your own custom trace viewer so you can remove friction and customize the display for your domain’s quirks. Take notes on problems you find. Do error analysis: categorize failures, figure out what to prioritize, decide what to work on.
\n![\"\"](\"https://hamel.dev/blog/posts/revenge/images/slide_13.png\")
When you look at your data, you end up driving toward application-specific metrics. Off-the-shelf similarity metrics like ROUGE or BLEU rarely fit LLM outputs. The metrics that matter look like “Calendar Scheduling Failure” or “Failure to Escalate To Human.”
\n![\"\"](\"https://hamel.dev/blog/posts/revenge/images/slide_14.png\")
If there is one thing to take away from this post: look at the data. How to look at it is a separate question and takes practice. This is the higest ROI activity you can engage in and is often skipped.
\nThe second pitfall is unverified judges. A lot of teams use an LLM as a judge to figure out whether their AI is working. Most of the time, nobody has a good answer to “how do you trust the judge?”
\n![\"\"](\"https://hamel.dev/blog/posts/revenge/images/slide_15.png\")
The default: ask an LLM to rate outputs on a scale and use the numbers. A data scientist would treat the judge like a classifier. You have a black box giving you a prediction. How do you trust it? Get human labels, partition the data into train/dev/test, and measure whether the classifier is trustworthy.
\n![\"\"](\"https://hamel.dev/blog/posts/revenge/images/slide_18.png\")
Source few-shot examples from your training set. Hill-climb your judge’s prompt against a dev set. Keep a test set aside to confirm you haven’t overfit. If you have done machine learning before, this is boring. But people are not doing it. Verifying classifiers has become a lost art in modern AI.
\n![\"\"](\"https://hamel.dev/blog/posts/revenge/images/slide_19.png\")
Treat your judge like a classifier in how you report results, too. Everywhere I go I see accuracy reported. If a failure mode occurs 5% of the time, accuracy hides the system’s true performance. Use precision and recall.
\nThe third pitfall is experimental design. There are many dimensions to this. Here are two that come up most.
\n![\"\"](\"https://hamel.dev/blog/posts/revenge/images/slide_20.png\")
The first is constructing test sets. Most teams generate synthetic data by prompting an LLM: “Give me 50 test queries.” They get generic, unrepresentative data. A data scientist would look at real production data first, use hypotheses to determine which dimensions matter, then generate synthetic examples along those dimensions.
\n![\"\"](\"https://hamel.dev/blog/posts/revenge/images/slide_23.png\")
Ground synthetic data in real logs or traces. Figure out what dimensions to vary. Inject edge cases. Base the synthetic data off real data.
\n![\"\"](\"https://hamel.dev/blog/posts/revenge/images/slide_26.png\")
The second is metric design. Teams bundle entire rubrics into a single LLM call and default to 1-5 Likert scales. A data scientist would reduce complexity, make each metric actionable, and tie it to a business outcome. Replace subjective scales with binary pass/fail on scoped criteria. Likert scales hide ambiguity and kick the can down the road on hard decisions about system performance.
\nThe fourth pitfall is bad data and labels. Data scientists don’t trust the data. They don’t trust the labels. They don’t trust anything. They are skeptical by training. AI engineers at large have not built this muscle yet.
\n![\"\"](\"https://hamel.dev/blog/posts/revenge/images/slide_27.png\")
When it comes to labeling, most teams make it someone else’s problem. Labeling seems unglamorous, so it gets delegated to the dev team or outsourced. A data scientist would insist that domain experts label the data, stay skeptical of the labels, and look at the data.
\n![\"\"](\"https://hamel.dev/blog/posts/revenge/images/slide_30.png\")
But labeling matters for a deeper reason than label quality. It is impossible to know what you want unless you look at the data. There is a concept called “criteria drift,” validated in a paper by Shreya Shankar and colleagues: users need criteria to grade outputs, but grading outputs helps users define their criteria. People don’t know what they want until they see the LLM’s outputs. The labeling process itself surfaces what matters.
\n![\"\"](\"https://hamel.dev/blog/posts/revenge/images/slide_31.png\")
Data scientists champion this: get domain experts and product managers in front of raw data, not summary scores.
\nThe fifth pitfall is automating too much. All of this is human work. The temptation is to automate it away.
\n![\"\"](\"https://hamel.dev/blog/posts/revenge/images/slide_32.png\")
![\"\"](\"https://hamel.dev/blog/posts/revenge/images/slide_33.png\")
LLMs can help wire things up, write the plumbing, generate boilerplate for evaluations. They cannot look at the data for you, for the exact reason we just discussed: you don’t know what you want until you see the outputs.
\nWe did not have time to cover every pitfall. Here is a speed run through the rest.
\n![\"\"](\"https://hamel.dev/blog/posts/revenge/images/slide_34.png\")
Misusing similarity scores. Asking the judge vague questions like “is it helpful?” Making annotators read raw JSON. Reporting uncalibrated scores without confidence intervals. Data drift, overfitting, not sampling correctly, dashboards that don’t make sense.
\nIf you zoom out, every pitfall above has the same root cause: missing a data science fundamental.
\n![\"\"](\"https://hamel.dev/blog/posts/revenge/images/slide_35.png\")
Reading traces and categorizing failures is Exploratory Data Analysis. Validating an LLM judge against human labels is Model Evaluation. Building representative test sets from production data is Experimental Design. Getting domain experts to label outputs is Data Collection. Monitoring whether your product works in production is Production ML. None of this is new!
\n![\"\"](\"https://hamel.dev/blog/posts/revenge/images/slide_36.png\")
This is a Python conference, so: Python remains the best toolset for looking at your data and dealing with data.
\n![\"\"](\"https://hamel.dev/blog/posts/revenge/images/slide_37.png\")
I built an evals skills plugin that goes into more depth. Point it at your eval pipeline and it will tell you what you are doing wrong (or try its best to).
\n![\"\"](\"https://hamel.dev/blog/posts/revenge/images/slide_38.png\")
Always look at the data.
\nIf you enjoyed the memes in this talk, there are many more on my website.
\nIf you want to go deeper on any of these topics, the slides and video are below.
\nThanks to Shreya Shankar and Bryan Bischof for many conversations that shaped this talk.
\nhttps://hbr.org/2012/10/data-scientist-the-sexiest-job-of-the-21st-century ↩
\nhttps://www.forbes.com/sites/louiscolumbus/2018/01/29/data-scientist-is-the-best-job-in-america-according-glassdoors-2018-rankings/ ↩
\nhttps://www.mckinsey.com/about-us/new-at-mckinsey-blog/ai-reinvents-tech-talent-opportunities ↩
\n![\"\"](\"https://hamel.dev/blog/posts/evals-skills/cover-original.png\")
Today, Shreya Shankar and I are publishing evals skills, a set of skills for AI product evals1.
\nEval tools often get in the way. They nudge you toward generic off-the-shelf metrics and fully automated evals before you’ve looked at your data. These skills help you avoid common mistakes we’ve seen helping 50+ companies and teaching students in our AI Evals course.
\nThere are many easily avoidable footguns in evals. These skills help you avoid them.
\nevals-start is the entry point. It looks at your situation and routes you to the right skill. Most of the time it will send you to one of these two:
\nInstall the skills:
\nnpx skills add https://github.com/ai-evals-course/evals-skills\nThen give your agent this prompt:
\n\n\nRun the evals-start skill from the evals plugin and follow the skill it picks. If it picks eval-audit, investigate each diagnostic area using a separate subagent in parallel, then synthesize the findings into a single report.
\n
If you’re experienced with evals, skip the router and pick the skill you need:
\n| Skill | \nWhat it does | \n
|---|---|
| evals-start | \nEntry point. Routes to the skill that matches your situation | \n
| eval-audit | \nAudit an eval pipeline and surface problems with prioritized severity | \n
| error-discovery | \nBuild a review app, select diverse samples, and organize your notes into failure modes | \n
| generate-synthetic-data | \nCreate diverse synthetic test inputs using dimension-based tuple generation | \n
| write-judge-prompt | \nDesign LLM-as-Judge evaluators for subjective quality criteria | \n
| validate-evaluator | \nCalibrate LLM judges against human labels using data splits, TPR/TNR, and bias correction | \n
| evaluate-rag | \nEvaluate retrieval and generation quality in RAG pipelines | \n
| build-review-interface | \nBuild custom annotation interfaces for human trace review | \n
These skills are only a starting point. To make them better, tune them to be more specific to your data and domain.
\nThe repo is ai-evals-course/evals-skills.
\nYou can find me on X or email me through my newsletter.
\nNot foundation model benchmarks like MMLU or HELM that measure general LLM capabilities. Product evals measure whether your pipeline works on your task with your data. If you aren’t familiar with product-specific AI evals, check out my AI Evals FAQ. ↩
\nThe audit isn’t a complete solution, but it will catch common problems. ↩
\nProgrammers love to proclaim they’ve found the best tool. Paul Graham called Lisp his “secret weapon.” DHH described Ruby as “a magical glove that just fit my brain perfectly.” Pieter Levels ships million-dollar products with vanilla PHP and jQuery.
\nThese declarations aren’t about the languages themselves. They’re about developers finding tools that fit how they think. When the environment clicks, you move fast.
\nI had that experience with nbdev, a development environment for literate programming that I helped build and maintain1. I created hundreds of projects with it and was one of its biggest proponents.
\nToday, I no longer use it. AI coding tools changed the trade-offs.
\nThe beauty of nbdev is its workflow. You write code, documentation and tests in one source of truth: Jupyter notebooks. Afterwards, these notebooks are transpiled into a Python library and documentation website.
\nThis workflow is idiosyncratic. AI coding tools, trained on vast amounts of conventional source code, get confused. They struggle to differentiate between editing the notebook and editing the final source code. It feels like fighting the AI instead of working with it.
\nI write software to solve problems, not to write code. I want to work in an environment where AI has the highest chance of success. With nbdev, I was swimming upstream.
\nSome argue that AI tools encourage lazy thinking: that without guardrails, developers skip the hard work of breaking problems into steps. But thinking step-by-step is a human skill. Notebooks don’t force you to write clean code. AI tools don’t force you to think carefully. Discipline comes from the developer, not the environment.
\nA central promise of literate programming is better documentation. By keeping code and docs in one place, you reduce the chance they become stale.
\nStrangely, many nbdev projects lacked sufficient documentation for my taste. Sometimes, this helped me learn a codebase by contributing to the docs. Other times, it was frustrating. This reinforced my belief that good documentation comes from effort, not tooling.
\nThis workflow is also less compelling now. AI can read a codebase without documentation and give you an overview on the fly. It can help maintain documentation that is separate from the code, handling the tedious parts. Keeping code and docs together isn’t the selling point it used to be.
\nnbdev asks developers to adopt a different system. It does not meet them where they are. Cursor won because it felt familiar and let developers change their habits slowly, rather than demanding a new workflow on day one.
\nI didn’t worry about collaboration as much before. But collaborating with AI is table stakes. The same impediments that get in the way of collaborating with humans tend to get in the way of collaborating with AI.
\nToday, developers increasingly span greater scope. Backend people do frontend. PMs create prototypes. Everyone is more polyglot. Idiosyncratic frameworks isolate you from your team to a greater extent than ever before. Idiosyncratic tooling once had a hidden upside: it filtered for a certain kind of contributor. Now I believe it’s more of a liability.
\nBecause I have invested thousands of hours into nbdev, it’s difficult to admit there are better tools for the outcomes I want. But I must check my ego at the door2.
\nMy daily drivers now include Amp, Cursor, and Claude Code. I still enjoy notebooks, but only for data analysis, machine learning, or other exploratory workflows where the iterative, visual nature of notebooks shines.
\nMore importantly, AI has nudged me out of my previous “Python for everything” mindset. I now use different languages for different tasks. For web development, I prefer the Next.js stack. Using a notebook for web development (even with specialized tooling) adds unnecessary complexity for me.
\nThis isn’t arbitrary preference. AI performs best on code with abundant training data for specific domains. TypeScript recently overtook both Python and JavaScript on GitHub, driven partly by the fact that typed languages make AI-generated code more reliable in production3. Even Jane Street, famous for its OCaml-heavy infrastructure, now uses Python for machine learning and data work.
\nNone of this means idiosyncratic tools are worthless. Lisp, Haskell, and APL each teach you something different about computing. Joy is a valid reason to choose a language, even with less AI support. It’s just not my focus right now. My joy resides in solving problems, and I want tools that maximize my leverage. For that, conventional wins.
\nI joined the nbdev project in 2020 while at GitHub. That same year I built fastpages, a notebook blogging system that informed nbdev’s documentation approach. Along the way I contributed to tools like ghapi and fastcore. In 2022, I helped lead a complete rewrite of nbdev. I discussed the philosophy behind this work on the Vanishing Gradients Podcast and at Data Council. I was briefly interested in commercializing nbdev, but decided not to pursue it. ↩
\nOther top nbdev maintainers and power users like Sylvain Gugger, Wasim Lorgat, Isaac Flath, Zach Mueller, and Wayde Gilliam have made similar moves. ↩
\nResearch shows 94% of LLM compilation errors in TypeScript come from type violations, suggesting type systems can guide better code generation. See Mündler et al., 2025. ↩
\nOver the past year, I’ve focused heavily on AI Evals, both in my consulting work and teaching. A question I get constantly is, “What’s the best tool for evals?”. I’ve always resisted answering directly for two reasons. First, people focus too much on tools instead of the process, thinking the tool will be an off-the-shelf solution when it rarely is. Second, the tools change so quickly that comparisons become outdated immediately.
\nHaving used many of the popular eval tools, I can genuinely say that no single one is superior in every dimension. The “best” tool depends on your team’s skillset, technical stack, and maturity.
\nInstead of a feature-by-feature comparison, I think it’s more valuable to show you how a panel of data scientists skilled in evals assesses these tools. As part of my AI Evals course, we had three of the most dominant vendors—Langsmith, Braintrust, and Arize Phoenix complete the same homework assignment. This gave us a unique opportunity to see how they tackle the exact same challenge.
\nWe recorded the entire process and live commentary, which is available below. We think this might be helpful in learning about the kinds of things you should consider when selecting a tool for your team.
\nThanks to Shreya Shankar and Bryan Bischof for serving as the panelists (alongside me).
\nWith Harrison Chase, CEO of LangChain.
\n\nWith Wayde Gilliam, former developer relations at Braintrust.
\n\nWith SallyAnn DeLucia, Technical AI Product Leader at Arize.
\n\nHere are themes that consistently surfaced during our review.
\nReducing friction is more important than any single feature. Concretely, you should be mindful of the time it takes to go from observing a failure to iterating on a solution. For example, we appreciated the ability to go from viewing a single trace to experimenting with that same trace in a playground. For some teams with data-science backgrounds, a notebook-centric workflow is ideal as it provides transparency and control. This happens to be my preferred workflow as well.
\nWhen considering a notebook-centric workflow, its important to pay attention to the ergonmics of the sdk. This often boils down to the quality of the documentation and integration with existing data tools.
\nThe best tools don’t try to automate away the human; they empower them. Since error analysis is the highest ROI activity in AI engineering, a tool’s ability to support efficient human review is paramount. Prioritize tools with first-class support for manual annotation and error analysis. As of this writing, one thing that is missing from many tools is axial coding.
\nBe deeply skeptical of features that promise full automation without human validation, as these can create a powerful and dangerous illusion of confidence. For example, be wary of features where an AI agent both creates an evaluation rubric and then immediately scores the outputs. This “stacking of abstractions” often hides flaws behind a high score. Favor tools that give you control and visibility.
\nAn eval tool should fit your stack, not force you to fit its stack. Assess how well a tool integrates with your existing technologies. Also, beware of proprietary DSLs as they can add friction. Finally, the ability to export data into common formats for analysis in a variety of environments is a must-have.
\nThe right choice of tool depends on your team’s workflow, skillset, and specific needs. I hope seeing how our panel approached this evaluation provides a better framework for making your own decision.
\nAs for me personally, I tend to use these tools as a backend data store and use Jupyter notebooks as well as my own custom built annotation interfaces for most of my needs.
\nCautionAppendix: Vendor Snapshots (As of Mid-2025)
\nYou should take these notes with a grain of salt. I recommend watch the videos above to get a sense of how we applied these criteria and where you might differ according to your neeeds.
\nOverall Sentiment The overall workflow is intuitive, especially for those new to formal evaluation processes. The UI guides you through creating datasets, running experiments, and annotating results.
\nPositive Feedback / What We Liked
\nCritiques and Areas for Improvement
\nOverall Sentiment The panel had a generally positive view of Braintrust, highlighting its clean UI and structured approach to evaluations. The tool’s emphasis on human-in-the-loop workflows was a significant strength.
\nPositive Feedback / What We Liked
\nCritiques and Areas for Improvement
\nOverall Sentiment The panel had a generally positive view of Phoenix, with one panelist calling it one of his “favorite open source eval tools.” The tool is positioned as a developer-first, notebook-centric platform.
\nPositive Feedback / What We Liked
\nCritiques and Areas for Improvement
\nMost AI teams focus on the wrong things. Here’s a common scene from my consulting work:
\nAI TEAM
\n\n\nHere’s our agent architecture – we’ve got RAG here, a router there, and we’re using this new framework for…
\n
ME
\n\n\n[Holding up my hand to pause the enthusiastic tech lead.]
\n“Can you show me how you’re measuring if any of this actually works?”
\n
… Room goes quiet
\nThis scene has played out dozens of times over the last two years. Teams invest weeks building complex AI systems, but can’t tell me if their changes are helping or hurting.
\nThis isn’t surprising. With new tools and frameworks emerging weekly, it’s natural to focus on tangible things we can control – which vector database to use, which LLM provider to choose, which agent framework to adopt. But after helping 30+ companies build AI products, I’ve discovered the teams who succeed barely talk about tools at all. Instead, they obsess over measurement and iteration.
\nIn this post, I’ll show you exactly how these successful teams operate. You’ll learn:
\nI’ll explain each of these topics with real examples. While every situation is unique, you’ll see patterns that apply regardless of your domain or team size.
\nLet’s start by examining the most common mistake I see teams make – one that derails AI projects before they even begin.
\nThe “tools first” mindset is the most common mistake in AI development. Teams get caught up in architecture diagrams, frameworks, and dashboards while neglecting the process of actually understanding what’s working and what isn’t.
\nOne client proudly showed me this evaluation dashboard:
\n![\"\"](\"https://hamel.dev/blog/posts/field-guide/images/dashboard.png\")
The kind of dashboard that foreshadows failure.
\nThis is the “tools trap” – the belief that adopting the right tools or frameworks (in this case, generic metrics) will solve your AI problems. Generic metrics are worse than useless – they actively impede progress in two ways:
\nFirst, they create a false sense of measurement and progress. Teams think they’re data-driven because they have dashboards, but they’re tracking vanity metrics that don’t correlate with real user problems. I’ve seen teams celebrate improving their “helpfulness score” by 10% while their actual users were still struggling with basic tasks. It’s like optimizing your website’s load time while your checkout process is broken – you’re getting better at the wrong thing.
\nSecond, too many metrics fragment your attention. Instead of focusing on the few metrics that matter for your specific use case, you’re trying to optimize multiple dimensions simultaneously. When everything is important, nothing is.
\nThe alternative? Error analysis - the single most valuable activity in AI development and consistently the highest-ROI activity. Let me show you what effective error analysis looks like in practice.
\nWhen Jacob, the founder of Nurture Boss, needed to improve their apartment-industry AI assistant, his team built a simple viewer to examine conversations between their AI and users. Next to each conversation was a space for open-ended notes about failure modes.
\nAfter annotating dozens of conversations, clear patterns emerged. Their AI was struggling with date handling – failing 66% of the time when users said things like “let’s schedule a tour two weeks from now.”
\nInstead of reaching for new tools, they: 1. Looked at actual conversation logs 2. Categorized the types of date-handling failures 3. Built specific tests to catch these issues 4. Measured improvement on these metrics
\nThe result? Their date handling success rate improved from 33% to 95%.
\nHere’s Jacob explaining this process himself:
\n\nWhen identifying error types, you can take either a “top-down” or “bottom-up” approach.
\nThe top-down approach starts with common metrics like “hallucination” or “toxicity” plus metrics unique to your task. While convenient, it often misses domain-specific issues.
\nThe more effective bottom-up approach forces you to look at actual data and let metrics naturally emerge. At NurtureBoss, we started with a spreadsheet where each row represented a conversation. We wrote open-ended notes on any undesired behavior. Then we used an LLM to build a taxonomy of common failure modes. Finally, we mapped each row to specific failure mode labels and counted the frequency of each issue.
\nThe results were striking - just three issues accounted for over 60% of all problems:
\n![\"\"](\"https://hamel.dev/blog/posts/field-guide/images/pivot.png\")
Excel Pivot Tables are a simple tool, but they work!
\nThe impact was immediate. Jacob’s team had uncovered so many actionable insights that they needed several weeks just to implement fixes for the problems we’d already found.
\nIf you’d like to see error analysis in action, we recorded a live walkthrough here.
\nThis brings us to a crucial question: How do you make it easy for teams to look at their data? The answer leads us to what I consider the most important investment any AI team can make…
\nThe single most impactful investment I’ve seen AI teams make isn’t a fancy evaluation dashboard – it’s building a customized interface that lets anyone examine what their AI is actually doing. I emphasize customized because every domain has unique needs that off-the-shelf tools rarely address. When reviewing apartment leasing conversations, you need to see the full chat history and scheduling context. For real estate queries, you need the property details and source documents right there. Even small UX decisions – like where to place metadata or which filters to expose – can make the difference between a tool people actually use and one they avoid.
\nI’ve watched teams struggle with generic labeling interfaces, hunting through multiple systems just to understand a single interaction. The friction adds up: clicking through to different systems to see context, copying error descriptions into separate tracking sheets, switching between tools to verify information. This friction doesn’t just slow teams down – it actively discourages the kind of systematic analysis that catches subtle issues.
\nTeams with thoughtfully designed data viewers iterate 10x faster than those without them. And here’s the thing: these tools can be built in hours using AI-assisted development (like Cursor or Loveable). The investment is minimal compared to the returns.
\nLet me show you what I mean. Here’s the data viewer built for NurtureBoss (which we discussed earlier):
\n![\"\"](\"https://hamel.dev/blog/posts/field-guide/images/nboss_filter.png\")
Search and filter sessions
\n![\"\"](\"https://hamel.dev/blog/posts/field-guide/images/nboss_annotate.png\")
Annotate and add notes
\n![\"\"](\"https://hamel.dev/blog/posts/field-guide/images/nboss_analysis.png\")
Aggregate and count errors
\nHere’s what makes a good data annotation tool:
\nIt doesn’t matter what web frameworks you use - use whatever you are familiar with. Because I’m a python developer, my current favorite web framework is FastHTML coupled with MonsterUI, because it allows me to define the back-end and front-end code in one small python file.
\nThe key is starting somewhere, even if it’s simple. I’ve found custom web apps provide the best experience, but if you’re just beginning, a spreadsheet is better than nothing. As your needs grow, you can evolve your tools accordingly.
\nThis brings us to another counter-intuitive lesson: the people best positioned to improve your AI system are often the ones who know the least about AI.
\nI recently worked with an education startup building an interactive learning platform with LLMs. Their product manager, a learning design expert, would create detailed PowerPoint decks explaining pedagogical principles and example dialogues. She’d present these to the engineering team, who would then translate her expertise into prompts.
\nBut here’s the thing: prompts are just English. Having a learning expert communicate teaching principles through PowerPoint, only for engineers to translate that back into English prompts, created unnecessary friction. The most successful teams flip this model by giving domain experts tools to write and iterate on prompts directly.
\nPrompt playgrounds are a great starting point for this. Tools like Arize, Langsmith and Braintrust let teams quickly test different prompts, feed in example datasets, and compare results. Here are some screenshots of these tools:
\n![\"\"](\"https://hamel.dev/blog/posts/field-guide/images/pp_phoenix2.png\")
Arize Phoenix
\n![\"\"](\"https://hamel.dev/blog/posts/field-guide/images/pp_langsmith.png\")
LangSmith
\n![\"\"](\"https://hamel.dev/blog/posts/field-guide/images/pp_bt.png\")
Braintrust
\nBut there’s a crucial next step that many teams miss: integrating prompt development into their application context. Most AI applications aren’t just prompts – They commonly involve RAG systems pulling from your knowledge base, agent orchestration coordinating multiple steps, and application-specific business logic. The most effective teams I’ve worked with go beyond standalone playgrounds. They build what I call integrated prompt environments – essentially admin versions of their actual user interface that expose prompt editing.
\nHere’s an illustration of what an integrated prompt environment might look like for a real estate AI assistant:
\n![\"\"](\"https://hamel.dev/blog/posts/field-guide/images/ipe_before.png\")
The UI that users (real estate agents) see.
\n![\"\"](\"https://hamel.dev/blog/posts/field-guide/images/ipe_after.png\")
The same UI, but with an “admin mode”used by the engineering & product team to iterate on the prompt and debug issues.
\nThere’s another barrier that often prevents domain experts from contributing effectively: unnecessary jargon. I was working with an education startup where engineers, product managers, and learning specialists were talking past each other in meetings. The engineers kept saying, “We’re going to build an agent that does XYZ,” when really the job to be done was writing a prompt. This created an artificial barrier – the learning specialists, who were the actual domain experts, felt like they couldn’t contribute because they didn’t understand “agents.”
\nThis happens everywhere. I’ve seen it with lawyers at legal tech companies, psychologists at mental health startups, and doctors at healthcare firms. The magic of LLMs is that they make AI accessible through natural language, but we often destroy that advantage by wrapping everything in technical terminology.
\nHere’s a simple example of how to translate common AI jargon:
\n| Instead of saying… | \nSay… | \n
|---|---|
| “We’re implementing a RAG approach” | \n“We’re making sure the model has the right context to answer questions” | \n
| “We need to prevent prompt injection” | \n“We need to make sure users can’t trick the AI into ignoring our rules” | \n
| “Our model suffers from hallucination issues” | \n“Sometimes the AI makes things up, so we need to check its answers” | \n
This doesn’t mean dumbing things down – it means being precise about what you’re actually doing. When you say “we’re building an agent,” what specific capability are you adding? Is it function calling? Tool use? Or just a better prompt? Being specific helps everyone understand what’s actually happening.
\nThere’s nuance here. Technical terminology exists for a reason – it provides precision when talking with other technical stakeholders. The key is adapting your language to your audience.
\nThe challenge many teams raise at this point is: “This all sounds great, but what if we don’t have any data yet? How can we look at examples or iterate on prompts when we’re just starting out?” That’s what we’ll talk about next.
\nOne of the most common roadblocks I hear from teams is: “We can’t do proper evaluation because we don’t have enough real user data yet.” This creates a chicken-and-egg problem – you need data to improve your AI, but you need a decent AI to get users who generate that data.
\nFortunately, there’s a solution that works surprisingly well: synthetic data. LLMs can generate realistic test cases that cover the range of scenarios your AI will encounter.
\nAs I wrote in my LLM-as-a-Judge blog post, synthetic data can be remarkably effective for evaluation. Bryan Bischof, the former Head of AI at Hex, put it perfectly:
\n\n\n“LLMs are surprisingly good at generating excellent - and diverse - examples of user prompts. This can be relevant for powering application features, and sneakily, for building Evals. If this sounds a bit like the Large Language Snake is eating its tail, I was just as surprised as you! All I can say is: it works, ship it.”
\n
The key to effective synthetic data is choosing the right dimensions to test. While these dimensions will vary based on your specific needs, I find it helpful to think about three broad categories:
\nThese aren’t the only dimensions you might care about – you might also want to test different tones of voice, levels of technical sophistication, or even different locales and languages. The important thing is identifying dimensions that matter for your specific use case.
\nFor a real estate CRM AI assistant I worked on with Rechat, we defined these dimensions like this:
\nfeatures = [\n "property search", # Finding listings matching criteria\n "market analysis", # Analyzing trends and pricing\n "scheduling", # Setting up property viewings\n "follow-up" # Post-viewing communication\n]\n\nscenarios = [\n "exact match", # One perfect listing match\n "multiple matches", # Need to help user narrow down\n "no matches", # Need to suggest alternatives\n "invalid criteria" # Help user correct search terms\n]\n\npersonas = [\n "first_time_buyer", # Needs more guidance and explanation\n "investor", # Focused on numbers and ROI\n "luxury_client", # Expects white-glove service\n "relocating_family" # Has specific neighborhood/school needs\n]\nBut having these dimensions defined is only half the battle. The real challenge is ensuring your synthetic data actually triggers the scenarios you want to test. This requires two things:
\nFor Rechat, we maintained a test database of listings that we knew would trigger different edge cases. Some teams prefer to use an anonymized copy of production data, but either way, you need to ensure your test data has enough variety to exercise the scenarios you care about.
\nHere’s an example of how we might use these dimensions with real data to generate test cases for the property search feature (this is just pseudo-code, and very illustrative):
\ndef generate_search_query(scenario, persona, listing_db):\n """Generate a realistic user query about listings"""\n # Pull real listing data to ground the generation\n sample_listings = listing_db.get_sample_listings(\n price_range=persona.price_range,\n location=persona.preferred_areas\n )\n \n # Verify we have listings that will trigger our scenario\n if scenario == "multiple_matches" and len(sample_listings) < 2:\n raise ValueError("Need multiple listings for this scenario")\n if scenario == "no_matches" and len(sample_listings) > 0:\n raise ValueError("Found matches when testing no-match scenario")\n \n prompt = f"""\n You are an expert real estate agent who is searching for listings. You are given a customer type and a scenario.\n \n Your job is to generate a natural language query you would use to search these listings.\n \n Context:\n - Customer type: \n - Scenario: \n \n Use these actual listings as reference:\n \n \n The query should reflect the customer type and the scenario.\n\n Example query: Find homes in the 75019 zip code, 3 bedrooms, 2 bathrooms, price range $750k - $1M for an investor.\n """\n return generate_with_llm(prompt)\nThis produced realistic queries like:
\n| Feature | \nScenario | \nPersona | \nGenerated Query | \n
|---|---|---|---|
| property search | \nmultiple matches | \nfirst_time_buyer | \n“Looking for 3-bedroom homes under $500k in the Riverside area. Would love something close to parks since we have young kids.” | \n
| market analysis | \nno matches | \ninvestor | \n“Need comps for 123 Oak St. Specifically interested in rental yield comparison with similar properties in a 2-mile radius.” | \n
The key to useful synthetic data is grounding it in real system constraints. For the real-estate AI assistant, this means:
\nWe then feed these test cases through Lucy and log the interactions. This gives us a rich dataset to analyze, showing exactly how the AI handles different situations with real system constraints. This approach helped us fix issues before they affected real users.
\nSometimes you don’t have access to a production database, especially for new products. In these cases, use LLMs to generate both test queries and the underlying test data. For a real estate AI assistant, this might mean creating synthetic property listings with realistic attributes – prices that match market ranges, valid addresses with real street names, and amenities appropriate for each property type. The key is grounding synthetic data in real-world constraints to make it useful for testing. The specifics of generating robust synthetic databases are beyond the scope of this post.
\nWhen generating synthetic data, follow these key principles to ensure it’s effective:
\nDiversify your dataset: Create examples that cover a wide range of features, scenarios, and personas. As I wrote in my LLM-as-a-Judge post, this diversity helps you identify edge cases and failure modes you might not anticipate otherwise.
\nGenerate user inputs, not outputs: Use LLMs to generate realistic user queries or inputs, not the expected AI responses. This prevents your synthetic data from inheriting the biases or limitations of the generating model.
\nIncorporate real system constraints: Ground your synthetic data in actual system limitations and data. For example, when testing a scheduling feature, use real availability windows and booking rules.
\nVerify scenario coverage: Ensure your generated data actually triggers the scenarios you want to test. A query intended to test “no matches found” should actually return zero results when run against your system.
\nStart simple, then add complexity: Begin with straightforward test cases before adding nuance. This helps isolate issues and establish a baseline before tackling edge cases.
\nThis approach isn’t just theoretical – it’s been proven in production across dozens of companies. What often starts as a stopgap measure becomes a permanent part of the evaluation infrastructure, even after real user data becomes available.
\nLet’s look at how to maintain trust in your evaluation system as you scale…
\nThis is a pattern I’ve seen repeatedly: teams build evaluation systems, then gradually lose faith in them. Sometimes it’s because the metrics don’t align with what they observe in production. Other times, it’s because the evaluations become too complex to interpret. Either way, the result is the same – the team reverts to making decisions based on gut feeling and anecdotal feedback, undermining the entire purpose of having evaluations.
\nMaintaining trust in your evaluation system is just as important as building it in the first place. Here’s how the most successful teams approach this challenge:
\nOne of the most insidious problems in AI evaluation is “criteria drift” – a phenomenon where evaluation criteria evolve as you observe more model outputs. In their paper “Who Validates the Validators?”, Shankar et al. describe this phenomenon:
\n\n\n“To grade outputs, people need to externalize and define their evaluation criteria; however, the process of grading outputs helps them to define that very criteria.”
\n
This creates a paradox: you can’t fully define your evaluation criteria until you’ve seen a wide range of outputs, but you need criteria to evaluate those outputs in the first place. In other words, it is impossible to completely determine evaluation criteria prior to human judging of LLM outputs.
\nI’ve observed this firsthand when working with Phillip Carter at Honeycomb on their Query Assistant feature. As we evaluated the AI’s ability to generate database queries, Phillip noticed something interesting:
\n\n\n“Seeing how the LLM breaks down its reasoning made me realize I wasn’t being consistent about how I judged certain edge cases.”
\n
The process of reviewing AI outputs helped him articulate his own evaluation standards more clearly. This isn’t a sign of poor planning – it’s an inherent characteristic of working with AI systems that produce diverse and sometimes unexpected outputs.
\nThe teams that maintain trust in their evaluation systems embrace this reality rather than fighting it. They treat evaluation criteria as living documents that evolve alongside their understanding of the problem space. They also recognize that different stakeholders might have different (sometimes contradictory) criteria, and they work to reconcile these perspectives rather than imposing a single standard.
\nSo how do you build evaluation systems that remain trustworthy despite criteria drift? Here are the approaches I’ve found most effective:
\nAs I wrote in my LLM-as-a-Judge post, binary decisions provide clarity that more complex scales often obscure. When faced with a 1-5 scale, evaluators frequently struggle with the difference between a 3 and a 4, introducing inconsistency and subjectivity. What exactly distinguishes “somewhat helpful” from “helpful”? These boundary cases consume disproportionate mental energy and create noise in your evaluation data. And even when businesses use a 1-5 scale, they inevitably ask where to draw the line for “good enough” or to trigger intervention, forcing a binary decision anyway.
\nIn contrast, a binary pass/fail forces evaluators to make a clear judgment: did this output achieve its purpose or not? This clarity extends to measuring progress – a 10% increase in passing outputs is immediately meaningful, while a 0.5-point improvement on a 5-point scale requires interpretation.
\nI’ve found that teams who resist binary evaluation often do so because they want to capture nuance. But nuance isn’t lost – it’s just moved to the qualitative critique that accompanies the judgment. The critique provides rich context about why something passed or failed, and what specific aspects could be improved, while the binary decision creates actionable clarity about whether improvement is needed at all.
\nWhile binary decisions provide clarity, they work best when paired with detailed critiques that capture the nuance of why something passed or failed. This combination gives you the best of both worlds: clear, actionable metrics and rich contextual understanding.
\nFor example, when evaluating a response that correctly answers a user’s question but contains unnecessary information, a good critique might read:
\n\n\n“The AI successfully provided the market analysis requested (PASS), but included excessive detail about neighborhood demographics that wasn’t relevant to the investment question. This makes the response longer than necessary and potentially distracting.”
\n
These critiques serve multiple functions beyond just explanation. They force domain experts to externalize implicit knowledge – I’ve seen legal experts move from vague feelings that something “doesn’t sound right” to articulating specific issues with citation formats or reasoning patterns that can be systematically addressed.
\nWhen included as few-shot examples in judge prompts, these critiques improve the LLM’s ability to reason about complex edge cases. I’ve found this approach often yields 15-20% higher agreement rates between human and LLM evaluations compared to prompts without example critiques. The critiques also provide excellent raw material for generating high-quality synthetic data, creating a flywheel for improvement.
\nIf you’re using LLMs to evaluate outputs (which is often necessary at scale), it’s crucial to regularly check how well these automated evaluations align with human judgment.
\nThis is particularly important given our natural tendency to over-trust AI systems. As Shankar et al. note in “Who Validates the Validators?”, the lack of tools to validate evaluator quality is concerning
\n\n\nResearch shows people tend to over-rely and over-trust AI systems. For instance, in one high profile incident, researchers from MIT posted a pre-print on arXiv claiming that GPT-4 could ace the MIT EECS exam. Within hours, [the] work [was] debunked … citing problems arising from over-reliance on GPT-4 to grade itself.”
\n
This over-trust problem extends beyond self-evaluation. Research has shown that LLMs can be biased by simple factors like the ordering of options in a set, or even seemingly innocuous formatting changes in prompts. Without rigorous human validation, these biases can silently undermine your evaluation system.
\nWhen working with Honeycomb, we tracked agreement rates between our LLM-as-a-judge and Phillip’s evaluations:
\n![\"\"](\"https://hamel.dev/blog/posts/field-guide/images/score.png\")
Agreement rates between LLM evaluator and human expert. More details here.
\nIt took three iterations to achieve >90% agreement, but this investment paid off in a system the team could trust. Without this validation step, automated evaluations often drift from human expectations over time, especially as the distribution of inputs changes. You can read more about this here.
\nTools like Eugene Yan’s AlignEval demonstrate this alignment process beautifully. It provides a simple interface where you upload data, label examples with a binary “good” or “bad,” and then evaluate LLM-based judges against those human judgments. What makes it effective is how it streamlines the workflow – you can quickly see where automated evaluations diverge from your preferences, refine your criteria based on these insights, and measure improvement over time. This approach reinforces that alignment isn’t a one-time setup but an ongoing conversation between human judgment and automated evaluation.
\nAs your AI system grows, you’ll inevitably face pressure to reduce the human effort involved in evaluation. This is where many teams go wrong – they automate too much, too quickly, and lose the human connection that keeps their evaluations grounded.
\nThe most successful teams take a more measured approach:
\nStart with high human involvement: In the early stages, have domain experts evaluate a significant percentage of outputs.
\nStudy alignment patterns: Rather than automating evaluation, focus on understanding where automated evaluations align with human judgment and where they diverge. This helps you identify which types of cases need more careful human attention.
\nUse strategic sampling: Rather than evaluating every output, use statistical techniques to sample outputs that provide the most information, particularly focusing on areas where alignment is weakest.
\nMaintain regular calibration: Even as you scale, continue to compare automated evaluations against human judgment regularly, using these comparisons to refine your understanding of when to trust automated evaluations.
\nScaling evaluation isn’t just about reducing human effort – it’s about directing that effort where it adds the most value. By focusing human attention on the most challenging or informative cases, you can maintain quality even as your system grows.
\nNow that we’ve covered how to maintain trust in your evaluations, let’s talk about a fundamental shift in how you should approach AI development roadmaps…
\nIf you’ve worked in software development, you’re familiar with traditional roadmaps: a list of features with target delivery dates. Teams commit to shipping specific functionality by specific deadlines, and success is measured by how closely they hit those targets.
\nThis approach fails spectacularly with AI.
\nI’ve watched teams commit to roadmaps like “Launch sentiment analysis by Q2” or “Deploy agent-based customer support by end of year,” only to discover that the technology simply isn’t ready to meet their quality bar. They either ship something subpar to hit the deadline or miss the deadline entirely. Either way, trust erodes.
\nThe fundamental problem is that traditional roadmaps assume we know what’s possible. With conventional software, that’s often true – given enough time and resources, you can build most features reliably. With AI, especially at the cutting edge, you’re constantly testing the boundaries of what’s feasible.
\nBryan Bischof, Former Head of AI at Hex, introduced me to what he calls a “capability funnel” approach to AI roadmaps. This strategy reframes how we think about AI development progress.
\nInstead of defining success as shipping a feature, the capability funnel breaks down AI performance into progressive levels of utility. At the top of the funnel is the most basic functionality – can the system respond at all? At the bottom is fully solving the user’s job to be done. Between these points are various stages of increasing usefulness.
\nFor example, in a query assistant, the capability funnel might look like: 1. Can generate syntactically valid queries (basic functionality) 2. Can generate queries that execute without errors 3. Can generate queries that return relevant results 4. Can generate queries that match user intent 5. Can generate optimal queries that solve the user’s problem (complete solution)
\nThis approach acknowledges that AI progress isn’t binary – it’s about gradually improving capabilities across multiple dimensions. It also provides a framework for measuring progress even when you haven’t reached the final goal.
\nThe most successful teams I’ve worked with structure their roadmaps around experiments rather than features. Instead of committing to specific outcomes, they commit to a cadence of experimentation, learning, and iteration.
\nEugene Yan, an applied scientist at Amazon, shared how he approaches ML project planning with leadership - a process that, while originally developed for traditional machine learning, applies equally well to modern LLM development:
\n\n\n“Here’s a common timeline. First, I take two weeks to do a data feasibility analysis, i.e”do I have the right data?” […] Then I take an additional month to do a technical feasibility analysis, i.e “can AI solve this?” After that, if it still works I’ll spend six weeks building a prototype we can A/B test.”
\n
While LLMs might not require the same kind of feature engineering or model training as traditional ML, the underlying principle remains the same: time-box your exploration, establish clear decision points, and focus on proving feasibility before committing to full implementation. This approach gives leadership confidence that resources won’t be wasted on open-ended exploration, while giving the team the freedom to learn and adapt as they go.
\nThe key to making an experiment-based roadmap work is having robust evaluation infrastructure. Without it, you’re just guessing whether your experiments are working. With it, you can rapidly iterate, test hypotheses, and build on successes.
\nI saw this firsthand during the early development of GitHub Copilot. What most people don’t realize is that the team invested heavily in building sophisticated offline evaluation infrastructure. They created systems that could test code completions against a very large corpus of repositories on GitHub, leveraging unit tests that already existed in high-quality codebases as an automated way to verify completion correctness. This was a massive engineering undertaking – they had to build systems that could clone repositories at scale, set up their environments, run their test suites, and analyze the results, all while handling the incredible diversity of programming languages, frameworks, and testing approaches.
\nThis wasn’t wasted time—it was the foundation that accelerated everything. With solid evaluation in place, the team ran thousands of experiments, quickly identified what worked, and could say with confidence “this change improved quality by X%” instead of relying on gut feelings. While the upfront investment in evaluation feels slow, it prevents endless debates about whether changes help or hurt, and dramatically speeds up innovation later.
\nThe challenge, of course, is that executives often want certainty. They want to know when features will ship and what they’ll do. How do you bridge this gap?
\nThe key is to shift the conversation from outputs to outcomes. Instead of promising specific features by specific dates, commit to a process that will maximize the chances of achieving the desired business outcomes.
\nEugene shared how he handles these conversations:
\n\n\n“I try to reassure leadership with timeboxes. At the end of three months, if it works out, then we move it to production. At any step of the way, if it doesn’t work out, we pivot.”
\n
This approach gives stakeholders clear decision points while acknowledging the inherent uncertainty in AI development. It also helps manage expectations about timelines – instead of promising a feature in six months, you’re promising a clear understanding of whether that feature is feasible in three months.
\nBryan’s capability funnel approach provides another powerful communication tool. It allows teams to show concrete progress through the funnel stages, even when the final solution isn’t ready. It also helps executives understand where problems are occurring and make informed decisions about where to invest resources.
\nPerhaps the most counterintuitive aspect of this approach is the emphasis on learning from failures. In traditional software development, failures are often hidden or downplayed. In AI development, they’re the primary source of learning.
\nEugene operationalizes this at his organization through what he calls a “fifteen-five” – a weekly update that takes fifteen minutes to write and five minutes to read:
\n\n\n“In my fifteen-fives, I document my failures and my successes. Within our team, we also have weekly”no-prep sharing sessions” where we discuss what we’ve been working on and what we’ve learned. When I do this, I go out of my way to share failures.”
\n
This practice normalizes failure as part of the learning process. It shows that even experienced practitioners encounter dead ends, and it accelerates team learning by sharing those experiences openly. And by celebrating the process of experimentation rather than just the outcomes, teams create an environment where people feel safe taking risks and learning from failures.
\nSo what does an experiment-based roadmap look like in practice? Here’s a simplified example from a content moderation project Eugene worked on:
\n\n\n“I was asked to do content moderation. I said, ‘It’s uncertain whether we’ll meet that goal. It’s uncertain even if that goal is feasible with our data, or what machine learning techniques would work. But here’s my experimentation roadmap. Here are the techniques I’m gonna try, and I’m gonna update you at a two-week cadence.’”
\n
The roadmap didn’t promise specific features or capabilities. Instead, it committed to a systematic exploration of possible approaches, with regular check-ins to assess progress and pivot if necessary.
\nThe results were telling:
\n\n\n“For the first two to three months, nothing worked. […] And then [a breakthrough] came out. […] Within a month, that problem was solved. So you can see that in the first quarter or even four months, it was going nowhere. […] But then you can also see that all of a sudden, some new technology comes along, some new paradigm, some new reframing comes along that just [solves] 80% of [the problem].”
\n
This pattern – long periods of apparent failure followed by breakthroughs – is common in AI development. Traditional feature-based roadmaps would have killed the project after months of “failure,” missing the eventual breakthrough.
\nBy focusing on experiments rather than features, teams create space for these breakthroughs to emerge. They also build the infrastructure and processes that make breakthroughs more likely – data pipelines, evaluation frameworks, and rapid iteration cycles.
\nThe most successful teams I’ve worked with start by building evaluation infrastructure before committing to specific features. They create tools that make iteration faster and focus on processes that support rapid experimentation. This approach might seem slower at first, but it dramatically accelerates development in the long run by enabling teams to learn and adapt quickly.
\nThe key metric for AI roadmaps isn’t features shipped – it’s experiments run. The teams that win are those that can run more experiments, learn faster, and iterate more quickly than their competitors. And the foundation for this rapid experimentation is always the same: robust, trusted evaluation infrastructure that gives everyone confidence in the results.
\nBy reframing your roadmap around experiments rather than features, you create the conditions for similar breakthroughs in your own organization.
\nThroughout this post, I’ve shared patterns I’ve observed across dozens of AI implementations. The most successful teams aren’t the ones with the most sophisticated tools or the most advanced models – they’re the ones that master the fundamentals of measurement, iteration, and learning.
\nThe core principles are surprisingly simple:
\nLook at your data. Nothing replaces the insight gained from examining real examples. Error analysis consistently reveals the highest-ROI improvements.
\nBuild simple tools that remove friction. Custom data viewers that make it easy to examine AI outputs yield more insights than complex dashboards with generic metrics.
\nEmpower domain experts. The people who understand your domain best are often the ones who can most effectively improve your AI, regardless of their technical background.
\nUse synthetic data strategically. You don’t need real users to start testing and improving your AI. Thoughtfully generated synthetic data can bootstrap your evaluation process.
\nMaintain trust in your evaluations. Binary judgments with detailed critiques create clarity while preserving nuance. Regular alignment checks ensure automated evaluations remain trustworthy.
\nStructure roadmaps around experiments, not features. Commit to a cadence of experimentation and learning rather than specific outcomes by specific dates.
\nThese principles apply regardless of your domain, team size, or technical stack. They’ve worked for companies ranging from early-stage startups to tech giants, across use cases from customer support to code generation.
\nIf you’d like to explore these topics further, here are some resources that might help:
\nMy blog for more content on AI evaluation and improvement. My other posts dive into more technical detail on topics such as constructing effective LLM judges, implementing evaluation systems, and other aspects of AI development1. Also check out the blogs of Shreya Shankar and Eugene Yan who are also great sources of information on these topics.
\nI teach these methods with Shreya Shankar in AI Evals For Engineers & PMs. The course includes practical exercises and office hours.
\nI write more broadly about machine learning, AI, and software development. Some posts that expand on these topics include Your AI Product Needs Evals, Creating a LLM-as-a-Judge That Drives Business Results, and What We’ve Learned From A Year of Building with LLMs. You can see all my posts at hamel.dev. ↩
\n![\"Hamel](\"https://hamel.dev/hamel_transparent.png\")
I’m a machine learning engineer with 20+ years of experience. I’ve worked at Airbnb and GitHub, including early LLM research used by OpenAI for code understanding. I’ve also led and contributed to popular open-source machine-learning tools.
\nI’m currently working to bring data science back to AI: helping teams debug, analyze, and measure their systems. I call this “evals,” and it’s the focus of my teaching and writing.
\nI co-teach AI Evals for Engineers and PMs, with over 5,000 students from 500+ companies (including OpenAI, Anthropic, and Google). Check the page for upcoming cohorts.
\nFor answers to common questions about evals, read the AI Evals Guide.
\nI often share my experience building AI products. Below is a selection of my long-form writing.
\nNo matching items
\nThe best way to follow my activity is X (Twitter).
\n", "date_modified": "2024-12-13T08:00:00+00:00", "date_published": "2024-12-13T08:00:00+00:00", "external_url": "https://hamel.dev/", "id": "urn:aggr:item:d5032e4ab558b0134fef0cef3e328439965e089b", "image": "https://hamel.dev/quarto.png", "summary": "Transform your GitHub Jupyter notebooks into beautiful, readable web pages with a single URL change. No setup required.", "tags": [ "programming", "jupyter" ], "title": "nbsanity - Share Notebooks as Polished Web Pages in Seconds", "url": "https://aggr.aymericbeaumet.com/items/hamel-dev/2024-12-13-nbsanity-share-notebooks-as-polished-web-pages-in-seconds/" }, { "authors": [ { "name": "Hamel Husain" } ], "content_html": "People often find me through my writing on AI and tech. This creates an interesting pattern. Nearly every week, vendors reach out asking me to write about their products. While I appreciate their interest and love learning about new tools, I reserve my writing for topics that I have personal experience with.
\nOne conversation last week really stuck with me. A founder confided, “We can write the best content in the world, but we don’t have any distribution.” This hit home because I used to think the same way.
\nLet me share what works for reaching developers. Companies and individuals alike often skip the basics when trying to grow their audience. These are proven approaches I’ve seen succeed, both in my work and in others’ efforts to grow their audience in the AI space.
\nHere’s something surprising: few people take the time to thoughtfully engage with others’ work in our field. But when you do, amazing things happen naturally.
\nFor example, here are some recent posts I’ve enjoyed that present opportunities to engage with others:
\nIn the above examples, you could share how their ideas connect with what you’ve built. You could add additional case studies and real-world insights. If you deeply engage with someone’s work and add your insights, they often share your content with their audience. Not because you asked, but because you’ve added something meaningful to their work. Swyx has written a great post on how to do this effectively.
\nThe key is authenticity. Don’t do this just for marketing—do it because you’re genuinely interested in learning from others and building on their ideas. It’s not hard to find things to be excited about. I’m amazed by how few people take this approach. It’s both effective and fun.
\nI see too many folks blogging or posting once every few months and wondering why they’re not getting traction. Want to know what actually works? Look at Jason Liu. He grew his following from 500 to 30,000 followers by posting ~ 30 times every day for a year.
\nYou don’t have to post that often (I certainly don’t!), but consistency matters more than perfection. Finally, don’t just post into the void. Engage with others. When someone comments on your post, reply thoughtfully. When you see conversations where you can add value, provide helpful information.
\nFinally, don’t be discouraged if you don’t see results immediately. Here’s some advice from my friend (and prolific writer), Eugene Yan:
\n\n\nIn the beginning, when most people start writing, the output’s gonna suck. Harsh, but true—my first 100 posts or so were crap. But with practice, people can get better. But they have to be deliberate in wanting to practice and get better with each piece, and not just write for the sake of publishing something and tweeting about it. The Sam Parr course (see below) is a great example of deliberate practice on copywriting.
\n
This changed everything for me. I took Sam Parr’s copywriting course just 30 minutes a day for a week. Now I keep my favorite writing samples in a Claude project and reference them when I’m writing something important. Small improvements in how you communicate can make a huge difference in how your content lands.
\nOne thing Sam teaches is that big words don’t make you sound smart. Clear writing that avoids jargon is more effective. That’s why Sam teaches aiming for a 6th-grade reading level. This matters even more with AI, as AI loves to generate flowery language and long sentences. The Hemingway App can be helpful in helping you simplify your writing.1
\nThe struggle most people have with creating content is that it takes too much time. But it doesn’t have to if you build the right systems, especially with AI.
\nGetting this system right takes some upfront work, but the payoff is enormous. Start by installing a good voice-to-text app on your phone. I use either Superwhisper or VoicePal. VoicePal is great for prompting you to elaborate with follow-up questions. These tools let me capture ideas at their best. That’s usually when I’m walking outside or away from my computer. At my computer, I use Flow.
\nThe key is to carefully craft your first few pieces of content. These become examples for your prompts that teach AI your style and tone. Once you have high-quality examples, you can organize these (transcript, content) pairs and feed them to language models. The in-context learning creates remarkably aligned output that matches your writing style while maintaining the authenticity of your original thoughts.
\nFor example, I use this pipeline at Answer AI. We have started interviewing each other and using the recordings as grounding for blog posts. Our recent post about SolveIt shows this in action. The raw conversation is the foundation. Our workflow turns it into polished content.
\nI’ve also integrated this workflow into my meetings. Using CircleBack, my favorite AI note-taking app, I can automatically capture and process meeting discussions. You can set up workflows to send your meeting notes and transcripts to AI for processing. This turns conversations into content opportunities.
\nThe real power comes from having all these pieces working together. Voice capture, AI, and automation makes content creation fun and manageable.
\nThrough my consulting work, I notice patterns that others miss. My most popular posts address common problems my clients had. When everyone’s confused about a topic, especially in AI where there’s lots of hype, clear explanations are gold. This is the motivation for some of my blog posts like:
\nYou probably see patterns too. Maybe it’s common questions from customers, or problems you’ve solved repeatedly. Maybe you work with a unique set of technologies or interesting use cases. Share these insights! Your unique perspective is more valuable than you think.
\nThis is probably the least important part of the process, but it’s still important. Thumbnails and social cards are vital for visibility on social media. Here are the tools I use:
\nFinally, when posting on X, threads can be a great way to raise the visibility of your content. A simple approach is to take screenshots or copy-paste snippets of your content. Then, walk through them in a thread, as you would want a reader to. Jeremy Howard does a great job at this: example 1, example 2.
\nOnce you have these systems in place, something magical happens: content creates more content. Your blog posts spawn social media updates. Your conversations turn into newsletters. Your client solutions become case studies. Each piece of work feeds the next, creating a natural flywheel.
\nDon’t try to sell too hard. Instead, share real insights and helpful information. Focus on adding value and educating your audience. When you do this well, people will want to follow your work.
\nThis journey is different for everyone. These are just the patterns I’ve seen work in my consulting practice and my own growth. Try what feels right. Adjust what doesn’t.
\nP.S. If you’d like to follow my writing journey, you can stay connected here.
\nDon’t abuse these tools or use them blindly. There’s plenty of situations where you should not be writing at a 6th-grade reading level. This includes, humor, poetry, shitposting, and more. Even formal writing shouldn’t adhere strictly to this rule. It’s advice that you should judge on a case-by-case basis. When you simplify your writing - do you like it more? ↩
\nEarlier this year, I wrote Your AI product needs evals. Many of you asked, “How do I get started with LLM-as-a-judge?” This guide shares what I’ve learned after helping over 30 companies set up their evaluation systems.
\nEver spend weeks building an AI system, only to realize you have no idea if it’s actually working? You’re not alone. I’ve noticed teams repeat the same mistakes when using LLMs to evaluate AI outputs:
\nThe result? Teams end up buried under mountains of metrics or data they don’t trust and can’t use. Progress grinds to a halt. Everyone gets frustrated.
\nFor example, it’s not uncommon for me to see dashboards that look like this:
\n![\"\"](\"https://hamel.dev/blog/posts/llm-judge/blog_header.png\")
An illustrative example of a bad eval dashboard
\nTracking a bunch of scores on a 1-5 scale is often a sign of a bad eval process (I’ll discuss why later). In this post, I’ll show you how to avoid these pitfalls. The solution is to use a technique that I call “Critique Shadowing”. Here’s how to do it, step by step.
\nIn most organizations there is usually one (maybe two) key individuals whose judgment is crucial for the success of your AI product. These are the people with deep domain expertise or represent your target users. Identifying and involving this Principal Domain Expert early in the process is critical.
\nWhy is finding the right domain expert so important?
\nThey Set the Standard: This person not only defines what is acceptable technically, but also helps you understand if you’re building something users actually want.
\nCapture Unspoken Expectations: By involving them, you uncover their preferences and expectations, which they might not be able to fully articulate upfront. Through the evaluation process, you help them clarify what a “passable” AI interaction looks like.
\nConsistency in Judgment: People in your organization may have different opinions about the AI’s performance. Focusing on the principal expert ensures that evaluations are consistent and aligned with the most critical standards.
\nSense of Ownership: Involving the expert gives them a stake in the AI’s development. They feel invested because they’ve had a hand in shaping it. In the end, they are more likely to approve of the AI.
\nExamples of Principal Domain Experts:
\nNoteExceptions
\nIn a smaller company, this might be the CEO or founder. If you are an independent developer, you should be the domain expert (but be honest with yourself about your expertise).
\nIf you must rely on leadership, you should regularly validate their assumptions against real user feedback.
\nMany developers attempt to act as the domain expert themselves, or find a convenient proxy (ex: their superior). This is a recipe for disaster. People will have varying opinions about what is acceptable, and you can’t make everyone happy. What’s important is that your principal domain expert is satisfied.
\nRemember: This doesn’t have to take a lot of the domain expert’s time. Later in this post, I’ll discuss how you can make the process efficient. Their involvement is absolutely critical to the AI’s success.
\nOnce you’ve found your expert, we need to give them the right data to review. Let’s talk about how to do that next.
\nWith your principal domain expert on board, the next step is to build a dataset that captures problems that your AI will encounter. It’s important that the dataset is diverse and represents the types of interactions that your AI will have in production.
\nYou want to define dimensions that make sense for your use case. For example, here are ones that I often use for B2C applications:
\n| Feature | \nDescription | \n
|---|---|
| Email Summarization | \nCondensing lengthy emails into key points. | \n
| Meeting Scheduler | \nAutomating the scheduling of meetings across time zones. | \n
| Order Tracking | \nProviding shipment status and delivery updates. | \n
| Contact Search | \nFinding and retrieving contact information from a database. | \n
| Language Translation | \nTranslating text between languages. | \n
| Content Recommendation | \nSuggesting articles or products based on user interests. | \n
Scenarios are situations the AI needs to handle, (not based on the outcome of the AI’s response).
\n| Scenario | \nDescription | \n
|---|---|
| Multiple Matches Found | \nUser’s request yields multiple results that need narrowing down. For example: User asks “Where’s my order?” but has three active orders (#123, #124, #125). AI must help identify which specific order they’re asking about. | \n
| No Matches Found | \nUser’s request yields no results, requiring alternatives or corrections. For example: User searches for order #ABC-123 which doesn’t exist. AI should explain valid order formats and suggest checking their confirmation email. | \n
| Ambiguous Request | \nUser input lacks necessary specificity. For example: User says “I need to change my delivery” without specifying which order or what aspect of delivery (date, address, etc.) they want to change. | \n
| Invalid Data Provided | \nUser provides incorrect data type or format. For example: User tries to track a return using a regular order number instead of a return authorization (RMA) number. | \n
| System Errors | \nTechnical issues prevent normal operation. For example: While looking up an order, the inventory database is temporarily unavailable. AI needs to explain the situation and provide alternatives. | \n
| Incomplete Information | \nUser omits required details. For example: User wants to initiate a return but hasn’t provided the order number or reason. AI needs to collect this information step by step. | \n
| Unsupported Feature | \nUser requests functionality that doesn’t exist. For example: User asks to change payment method after order has shipped. AI must explain why this isn’t possible and suggest alternatives. | \n
| Persona | \nDescription | \n
|---|---|
| New User | \nUnfamiliar with the system; requires guidance. | \n
| Expert User | \nExperienced; expects efficiency and advanced features. | \n
| Non-Native Speaker | \nMay have language barriers; uses non-standard expressions. | \n
| Busy Professional | \nValues quick, concise responses; often multitasking. | \n
| Technophobe | \nUncomfortable with technology; needs simple instructions. | \n
| Elderly User | \nMay not be tech-savvy; requires patience and clear guidance. | \n
This taxonomy (features, scenarios, personas) is not universal. For example, it may not make sense to even have personas if users aren’t directly engaging with your AI. The idea is you should outline dimensions that make sense for your use case and generate data that covers them. You’ll likely refine these after the first round of evaluations.
\nTo build your dataset, you can:
\nOften, you’ll do a combination of both to ensure comprehensive coverage. Synthetic data is not as good as real data, but it’s a good starting point. Also, we are only using LLMs to generate the user inputs, not the LLM responses or internal system behavior.
\nRegardless of whether you use existing data or synthetic data, you want good coverage across the dimensions you’ve defined.
\nIncorporating System Information
\nWhen making test data, use your APIs and databases where appropriate. This will create realistic data and trigger the right scenarios. Sometimes you’ll need to write simple programs to get this information. That’s what the “Assumptions” column is referring to in the examples below.
\nHere are some example prompts that illustrate how to use an LLM to generate synthetic user inputs for different combinations of features, scenarios, and personas:
\n| ID | \nFeature | \nScenario | \nPersona | \nLLM Prompt to Generate User Input | \nAssumptions (not directly in the prompt) | \n
|---|---|---|---|---|---|
| 1 | \nOrder Tracking | \nInvalid Data Provided | \nFrustrated Customer | \n“Generate a user input from someone who is clearly irritated and impatient, using short, terse language to demand information about their order status for order number #1234567890. Include hints of previous negative experiences.” | \nOrder number #1234567890 does not exist in the system. | \n
| 2 | \nContact Search | \nMultiple Matches Found | \nNew User | \n“Create a user input from someone who seems unfamiliar with the system, using hesitant language and asking for help to find contact information for a person named ‘Alex’. The user should appear unsure about what information is needed.” | \nMultiple contacts named ‘Alex’ exist in the system. | \n
| 3 | \nMeeting Scheduler | \nAmbiguous Request | \nBusy Professional | \n“Simulate a user input from someone who is clearly in a hurry, using abbreviated language and minimal details to request scheduling a meeting. The message should feel rushed and lack specific information.” | \nN/A | \n
| 4 | \nContent Recommendation | \nNo Matches Found | \nExpert User | \n“Produce a user input from someone who demonstrates in-depth knowledge of their industry, using specific terminology to request articles on sustainable supply chain management. Use the information in this article involving sustainable supply chain management to formulate a plausible query: {{article}}” | \nNo articles on ‘Emerging trends in sustainable supply chain management’ exist in the system. | \n
When generating synthetic data, you only need to create the user inputs. You then feed these inputs into your AI system to generate the AI’s responses. It’s important that you log everything so you can evaluate your AI. To recap, here’s the process:
\nThere is no right answer here. At a minimum, you want to generate enough data so that you have examples for each combination of dimensions (in this toy example: features, scenarios, and personas). However, you also want to keep generating more data until you feel like you have stopped seeing new failure modes. The amount of data I generate varies significantly depending on the use case.
\nYou might be skeptical of using synthetic data. After all, it’s not real data, so how can it be a good proxy? In my experience, it works surprisingly well. Some of my favorite AI products, like Hex use synthetic data to power their evals:
\n\n\n“LLMs are surprisingly good at generating excellent - and diverse - examples of user prompts. This can be relevant for powering application features, and sneakily, for building Evals. If this sounds a bit like the Large Language Snake is eating its tail, I was just as surprised as you! All I can say is: it works, ship it.” Bryan Bischof, Head of AI Engineering at Hex
\n
With your dataset ready, now comes the most important part: getting your principal domain expert to evaluate the interactions.
\nThe domain expert’s job is to focus on one thing: “Did the AI achieve the desired outcome?” No complex scoring scales or multiple metrics. Just a clear pass or fail decision. In addition to the pass/fail decision, the domain expert should write a critique that explains their reasoning.
\nClarity and Focus: A binary decision forces everyone to consider what truly matters. It simplifies the evaluation to a single, crucial question.
\nActionable Insights: Pass/fail judgments are easy to interpret and act upon. They help you quickly identify whether the AI meets the user’s needs.
\nForces Articulation of Expectations: When domain experts must decide if an interaction passes or fails, they are compelled to articulate their expectations clearly. This process uncovers nuances and unspoken assumptions about how the AI should behave.
\nEfficient Use of Resources: Keeps the evaluation process manageable, especially when starting out. You avoid getting bogged down in detailed metrics that might not be meaningful yet.
\nAlongside a binary pass/fail judgment, it’s important to write a detailed critique of the LLM-generated output. These critiques:
\nCaptures Nuances: The critique allows you to note if something was mostly correct but had areas for improvement.
\nGuide Improvement: Detailed feedback provides specific insights into how the AI can be enhanced.
\nBalance Simplicity with Depth: While the pass/fail offers a clear verdict, the critique offers the depth needed to understand the reasoning behind the judgment.
\nWhy Write Critiques?:
\nIn practice, domain experts may not have fully internalized all the judgment criteria. By forcing them to make a pass/fail decision and explain their reasoning, they clarify their expectations and provide valuable guidance for refining the AI.
\n“But my problem is complex!” Trust me—starting simple forces you to focus on what truly matters. You can introduce more complexity later if needed.
\nTo illustrate how simple pass/fail judgments combined with detailed critiques work in practice, here’s a table showcasing examples of user interactions with an AI assistant. The table includes both passes and fails, with critiques explaining why the AI received that judgment. In cases where the AI passed despite critical issues, the critique highlights these aspects and justifies why it still passed overall. For failed interactions, the critique explains the critical elements that led to the failure.
\n| User Interaction with AI | \nJudgment | \nCritique | \n
|---|---|---|
| User: “I need to cancel my flight booking for next week.” AI: “Your flight booking for next week has been canceled. A confirmation email has been sent to you.” | \nPass | \nThe AI successfully executed the user’s request by canceling the flight and sending a confirmation, achieving the primary goal. However, it should have asked for confirmation before proceeding to prevent accidental cancellations. Despite this critical oversight, the AI fulfilled the user’s request, so it passes, but adding a confirmation step would improve user safety. | \n
| User: “What’s the capital of France?” AI: “The capital of France is Paris.” | \nPass | \nThe AI correctly answered the user’s question, providing accurate information promptly. Although the response is sufficient, the AI could enhance the interaction by offering additional interesting facts or asking if the user has more questions. Despite this opportunity for added engagement, the AI met the user’s primary need, so it passes. | \n
| User: “I can’t log into my account; it says my password is incorrect.” AI: “Please reset your password by clicking ‘Forgot Password’.” | \nFail | \nThe AI provided a generic solution without acknowledging the user’s frustration or checking for other issues like account lockout. It failed to offer empathetic support or additional assistance. Due to the lack of personalized help and poor user experience, it fails. | \n
| User: “Book a table for two at an Italian restaurant tonight at 7 PM.” AI: “Table for two at Luigi’s Italian Restaurant booked for tonight at 7 PM. Confirmation number is 12345.” | \nPass | \nThe AI efficiently completed the booking as requested, providing all necessary details and a confirmation number. However, it could have asked if the user had seating preferences or dietary restrictions. The AI accomplished the primary task successfully, so it passes, with minor opportunities for enhanced service. | \n
| User: “I need help with my billing statement.” AI: “Please provide your account number and full name.” | \nFail | \nThe AI requested sensitive information without verifying the user’s identity through secure means, potentially compromising security. It should guide the user through a secure authentication process first. Due to this critical oversight in user data protection, it fails. | \n
These examples demonstrate how the AI can receive both “Pass” and “Fail” judgments. In the critiques:
\nFor passes, we explain why the AI succeeded in meeting the user’s primary need, even if there were critical aspects that could be improved. We highlight these areas for enhancement while justifying the overall passing judgment.
\nFor fails, we identify the critical elements that led to the failure, explaining why the AI did not meet the user’s main objective or compromised important factors like user experience or security.
\nMost importantly, the critique should be detailed enough so that you can use it in a few-shot prompt for a LLM judge. In other words, it should be detailed enough that a new employee could understand it. Being too terse is a common mistake.
\nNote that the example user interactions with the AI are simplified for brevity - but you might need to give the domain expert more context to make a judgement. More on that later.
\nNote
\nAt this point, you don’t need to perform a root cause analysis into the technical reasons behind why the AI failed. Many times, it’s useful to get a sense of overall behavior before diving into the weeds.
\nA common mistake is straying from binary pass/fail judgments. Let’s revisit the dashboard from earlier:
\n![\"\"](\"https://hamel.dev/blog/posts/llm-judge/dashboard.png\")
If your evaluations consist of a bunch of metrics that LLMs score on a 1-5 scale (or any other scale), you’re doing it wrong. Let’s unpack why.
\nThis is why I hate off the shelf metrics that come with many evaluation frameworks. They tend to lead people astray.
\nCommon Objections to Pass/Fail Judgments:
\nI can guarantee you that if someone says you need to measure 8 things on a 1-5 scale, they don’t know what they are looking for. They are just guessing. You have to let the domain expert drive and make a pass/fail judgment with critiques so you can figure out what truly matters. Stand your ground here.
\nFinally, you need to remove all friction from reviewing data. I’ve written about this here. Sometimes, you can just use a spreadsheet. It’s a judgement call in terms of what is easiest for the domain expert. I found that I often have to provide additional context to help the domain expert understand the user interaction, such as:
\nAll of this data needs to be presented on a single screen so the domain expert can review it without jumping through hoops. That’s why I recommend building a simple web app to review data.
\nThe number of examples you need depends on the complexity of the task. My heuristic is that I start with around 30 examples and keep going until I do not see any new failure modes. From there, I keep going until I’m not learning anything new.
\nThat heuristic is for discovering failure modes and writing the rubric. Validating an automated judge needs a larger labeled set. Aim for about 100 examples per failure mode, with enough Pass and Fail examples to measure both classes. Below 60 examples, the confidence intervals are often too wide to support a useful conclusion.
\nNext, we’ll look at how to use this data to build an LLM judge.
\nAfter looking at the data, it’s likely you will find errors in your AI system. Instead of plowing ahead and building an LLM judge, you want to fix any obvious errors. Remember, the whole point of the LLM as a judge is to help you find these errors, so it’s totally fine if you find them earlier!
\nIf you have already developed Level 1 evals as outlined in my previous post, you should not have any pervasive errors. However, these errors can sometimes slip through the cracks. If you find pervasive errors, fix them and go back to step 3. Keep iterating until you feel like you have stabilized your system.
\nYou cannot write a good judge prompt until you’ve seen the data. The paper from Shankar et al., “Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences” summarizes this well:
\n\n\nto grade outputs,people need to externalize and define their evaluation criteria; however, the process of grading outputs helps them to define that very criteria. We dub this phenomenon criteria drift, and it implies thatit is impossible to completely determine evaluation criteria prior to human judging of LLM outputs.
\n
Let me share a real-world example of building an LLM judge you can apply to your own use case. When I was helping Honeycomb build their Query Assistant feature, we needed a way to evaluate if the AI was generating good queries. Here’s what our LLM judge prompt looked like, including few-shot examples of critiques from our domain expert, Phillip:
\nYou are a Honeycomb query evaluator with advanced capabilities to judge if a query is good or not.\nYou understand the nuances of the Honeycomb query language, including what is likely to be\nmost useful from an analytics perspective. \n\nHere is information about the Honeycomb query language:\n{{query_language_info}}\n\nHere are some guidelines for evaluating queries:\n{{guidelines}}\n\nExample evaluations:\n\n<examples>\n\n<example-1>\n<nlq>show me traces where ip is 10.0.2.90</nlq>\n<query>\n{\n "breakdowns": ["trace.trace_id"],\n "calculations": [{"op": "COUNT"}],\n "filters": [{"column": "net.host.ip", "op": "=", "value": "10.0.2.90"}]\n}\n</query>\n<critique>\n{\n "critique": "The query correctly filters for traces with an IP address of 10.0.2.90 \n and counts the occurrences of those traces, grouped by trace.trace_id. The response \n is good as it meets the requirement of showing traces from a specific IP address \n without additional complexities.",\n "outcome": "good"\n}\n</critique>\n</example-1>\n\n<example-2>\n<nlq>show me slowest trace</nlq>\n<query>\n{\n "calculations": [{"column": "duration_ms", "op": "MAX"}],\n "orders": [{"column": "duration_ms", "op": "MAX", "order": "descending"}],\n "limit": 1\n}\n</query>\n<critique>\n{\n "critique": "While the query attempts to find the slowest trace using MAX(duration_ms) \n and ordering correctly, it fails to group by trace.trace_id. Without this grouping, \n the query only shows the MAX(duration_ms) measurement over time, not the actual \n slowest trace.",\n "outcome": "bad"\n}\n</critique>\n</example-2>\n\n<example-3>\n<nlq>count window-hash where window-hash exists per hour</nlq>\n<query>\n{\n "breakdowns": ["window-hash"],\n "calculations": [{"op": "COUNT"}],\n "filters": [{"column": "window-hash", "op": "exists"}],\n "time_range": 3600\n}\n</query>\n<critique>\n{\n "critique": "While the query correctly counts window-hash occurrences, the time_range \n of 3600 seconds (1 hour) is insufficient for per-hour analysis. When we say 'per hour', \n we need a time_range of at least 36000 seconds to show meaningful hourly patterns.",\n "outcome": "bad"\n}\n</critique>\n</example-3>\n\n</examples>\n\nFor the following query, first write a detailed critique explaining your reasoning,\nthen provide a pass/fail judgment in the same format as above.\n\n<nlq>{{user_input}}</nlq>\n<query>\n{{generated_query}}\n</query>\n<critique>\nNotice how each example includes:
\n<nlq> tags<query> tags<critique> tagsIn the prompt above, the example critiques are fixed. An advanced approach is to include examples dynamically based upon the item you are judging. You can learn more in this post about Continual In-Context Learning.
\nIn this case, I used a low-tech approach to iterate on the prompt. I sent Phillip a spreadsheet with the following information:
\nPhillip would then fill out his own version of the spreadsheet with his critiques. I used this to iteratively improve the prompt. The spreadsheet looked like this:
\n![\"\"](\"https://hamel.dev/blog/posts/llm-judge/spreadsheet.png\")
I also tracked agreement rates over time to ensure we were converging on a good prompt.
\n![\"\"](\"https://hamel.dev/blog/posts/llm-judge/score.png\")
WarningImportant Note on Using Agreement as a Metric
\nIn this example, we used agreement between the model and human evaluator because our dataset was roughly balanced (about 50% of instances were failures). Raw agreement can be misleading when classes are imbalanced. Treat the human labels as ground truth and report the judge’s True Positive Rate and True Negative Rate separately.
\nIt took us only three iterations to achieve > 90% agreement between the LLM and Phillip. Your mileage may vary depending on the complexity of the task. For example, Swyx has conducted a similar process hundreds of times for AI News, an extremely popular news aggregator with high quality recommendations. The quality of the AI owing to this process is why this product has received critical acclaim.
\nI usually adjust the prompts by hand. I haven’t had much luck with prompt optimizers like DSPy. However, my friend Eugene Yan has just released a promising tool named ALIGN Eval. I like it because it’s simple and effective. Also, don’t forget the approach of continual in-context learning mentioned earlier - it can be effective when implemented correctly.
\nIn rare cases, I might fine-tune a judge, but I prefer not to. I talk about this more in the FAQ section.
\nSomething unexpected happened during this process. Phillip Carter, our domain expert at Honeycomb, found that reviewing the LLM’s critiques helped him articulate his own evaluation criteria more clearly. He said,
\n\n\n“Seeing how the LLM breaks down its reasoning made me realize I wasn’t being consistent about how I judged certain edge cases.”
\n
This is a pattern I’ve seen repeatedly—the process of building an LLM judge often helps standardize evaluation criteria.
\nFurthermore, because this process forces the domain expert to look at data carefully, I always uncover new insights about the product, AI capabilities, and user needs. The resulting benefits are often more valuable than creating a LLM judge!
\nI conduct this human review at regular intervals and whenever something material changes. For example, if I update a model, I’ll run the process again. I don’t get too scientific here; instead, I rely on my best judgment. Also note that after the first two iterations, I tend to focus more on errors rather than sampling randomly. For example, if I find an error, I’ll search for more examples that I think might trigger the same error. However, I always do a bit of random sampling as well.
\nI’ve seen this process fail when:
\nIn each of these cases, I try to address the root cause instead of trying to force alignment. Sometimes, you may not be able to achieve the alignment you want and may have to lean heavier on human annotations. However, after following the process described here, you will have metrics that help you understand how much you can trust the LLM judge.
\nMost of the mistakes I’ve seen in LLM judge prompts have to do with not providing good examples:
\nSometimes, you may encounter difficulties with fitting everything you need into the prompt, and may have to get creative about how you structure the examples. However, this is becoming less of an issue thanks to expanding context windows and prompt caching.
\nAfter you have created a LLM as a judge, you will have a dataset of user interactions with the AI, and the LLM’s judgments. If your metrics show an acceptable agreement between the domain expert and the LLM judge, you can apply this judge against real or synthetic interactions. After this, you can you calculate error rates for different dimensions of your data. You should calculate the error on unseen data only to make sure your aren’t getting biased results.
\nFor example, if you have segmented your data by persona, scenario, feature, etc, your data analysis may look like this
\nError Rates by Key Dimensions
\n| Feature | \nScenario | \nPersona | \nTotal Examples | \nFailure Rate | \n
|---|---|---|---|---|
| Order Tracking | \nMultiple Matches | \nNew User | \n42 | \n24.3% | \n
| Order Tracking | \nMultiple Matches | \nExpert User | \n38 | \n18.4% | \n
| Order Tracking | \nNo Matches | \nExpert User | \n30 | \n23.3% | \n
| Order Tracking | \nNo Matches | \nNew User | \n20 | \n75.0% | \n
| Contact Search | \nMultiple Matches | \nNew User | \n35 | \n22.9% | \n
| Contact Search | \nMultiple Matches | \nExpert User | \n32 | \n19.7% | \n
| Contact Search | \nNo Matches | \nNew User | \n25 | \n68.0% | \n
| Contact Search | \nNo Matches | \nExpert User | \n28 | \n21.4% | \n
Once you know where the errors are now you can perform an error analysis to get to the root cause of the errors. My favorite way is to look at examples of each type of error and classify them by hand. I recommend using a spreadsheet for this. For example, a trace for Order tracking where there are no matches for new users might look like this:
\nTipExample Trace
\nIn this example trace, the user provides an invalid order number. The AI correctly identifies that the order number is invalid but provides an unhelpful response. If you are not familiar with logging LLM traces, refer to my previous post on evals.
\nNote that this trace is formatted for readability.
\n\nIn this case, you might classify the error as: Missing User Education. The system retrieved new user context and format information but failed to include it in the response, which suggests we could improve our prompt. After you have classified a number of errors, you can calculate the distribution of errors by root cause. That might look like this:
Root Cause Distribution (20 Failed Interactions)
\n| Root Cause | \nCount | \nPercentage | \n
|---|---|---|
| Missing User Education | \n8 | \n40% | \n
| Authentication/Access Issues | \n6 | \n30% | \n
| Poor Context Handling | \n4 | \n20% | \n
| Inadequate Error Messages | \n2 | \n10% | \n
Now you know where to focus your efforts. This doesn’t have to take an extraordinary amount of time. You can get quite far in just 15 minutes. Also, you can use a LLM to help you with this classification, but that is beyond the scope of this post (you can use a LLM to help you do anything in this post, as long as you have a process to verify the results).
\nError analysis has been around in Machine Learning for quite some time. This video by Andrew Ng does a great job of walking through the process interactively:
\n\nNow that you have a sense of the errors, you can go back and fix them again. Go back to step 3 and iterate until you are satisfied. Note that every time you fix an error, you should try to write a test case for it. Sometimes, this can be an assertion in your test suite, but other times you may need to create a more “specialized” LLM judge for these failures. We’ll talk about this next.
\nInvestigating your data is much harder in practice than I made it look in this post. It requires a nose for data that only comes from practice. It also helps to have some basic familiarity with statistics and data analysis tools. My favorite post on data literacy is this one by Jason Liu and Eugene Yan.
\nNow that you have a sense for where the problems in your AI are, you can decide where and if to invest in more targeted LLM judges. For example, if you find that the AI has trouble with citing sources correctly, you can created a targeted eval for that. You might not even need a LLM judge for some errors (and use a code-based assertion instead).
\nThe key takeaway is don’t jump directly to using specialized LLM judges until you have gone through this critique shadowing process. This will help you rationalize where to invest your time.
\nUsing an LLM as a judge can streamline your AI evaluation process if approached correctly. Here’s a visual illustration of the process (there is a description of the process below the diagram as well):
\ngraph TB\n A[Start] --> B[1 Find Principal Domain Expert]\n B --> C[2 Create Dataset]\n C --> D[3 Domain Expert Reviews Data]\n D --> E{Found Errors?}\n E -->|Yes| F[4 Fix Errors]\n F --> D\n E -->|No| G[5 Build LLM Judge]\n G --> H[Test Against Domain Expert]\n H --> I{Acceptable Agreement?}\n I -->|No| J[Refine Prompt]\n J --> H\n I -->|Yes| K[6 Perform Error Analysis]\n K --> L{Critical Issues Found?}\n L -->|Yes| M[7 Fix Issues & Create Specialized Judges]\n M --> D\n L -->|No| N[Material Changes or Periodic Review?]\n N -->|Yes| C\nThe Critique Shadowing process is iterative, with feedback loops. Let’s list out the steps:
\nThis process never truly ends. It repeats periodically or when material changes occur.
\nThe real value of this process is looking at your data and doing careful analysis. Even though an AI judge can be a helpful tool, going through this process is what drives results. I would go as far as saying that creating a LLM judge is a nice “hack” I use to trick people into carefully looking at their data!
\nThat’s right. The real business value comes from looking at your data. But hey, potato, potahto.
\nPhew, this seems like a lot of work! Do you really need this? Well, it depends. There are cases where you can take a shortcut through this process. For example, let’s say:
\nIn this scenario, you can jump directly to something that looks like step 3 and start looking at data right away. Also, since it’s not that costly to look at data, it’s probably fine to just do error analysis without a judge (at least initially). You can incorporate what you learn directly back into your primary model right away. This example is not exhaustive, but gives you an idea of how you can adapt this process to your needs.
\nHowever, you can never completely eliminate looking at your data! This is precisely the step that most people skip. Don’t be that person.
\nI received a lot of questions about this topic. Here are answers to the most common ones:
\nEffective judges often use larger models or more compute (via longer prompts, chain-of-thought, etc.) than the systems they evaluate.
\nHowever, If the cost of the most powerful LLM is not prohibitive, and latency is not an issue, then you might want to consider where you invest your efforts differently. In this case, it might make sense to put more effort towards specialist LLM judges, code-based assertions, and A/B testing. However, you should still go through the process of looking at data and critiquing the LLM’s output before you adopt specialized judges.
\nI prefer not to fine-tune LLM judges. I’d rather spend the effort fine-tuning the actual LLM instead. However, fine-tuning guardrails or other specialized judges can be useful (especially if they are small classifiers).
\nAs a related note, you can leverage a LLM judge to curate and transform data for fine-tuning your primary model. For example, you can use the judge to:
\nUsing a LLM judge for enhancing fine-tuning data is even more compelling when you are trying to distill a large LLM into a smaller one. The details of fine-tuning are beyond the scope of this post. If you are interested in learning more, see these resources.
\nNothing is strictly wrong with them. It’s just that many people are led astray by them. If you are disciplined you can apply them to your data and see if they are telling you something valuable. However, I’ve found that these tend to cause more confusion than value.
\nTreat the judge as a binary classifier. Ask a trusted domain expert to label examples, then split them into train, dev, and test sets.
\nA typical split uses 10 to 20 percent of the labeled examples for training, with 40 to 45 percent each for dev and test. The dev and test sets should each contain enough Pass and Fail examples to estimate both rates. Aim for 30 to 50 examples of each label in each set when possible. Never place dev or test examples in the judge prompt.
\nFinally, measure judge performance using true positive rate and true negative rate, since agreement can be misleading when failures are rare. For example, if an error occurs in 5% of examples, a judge that always predicts Pass still has 95% agreement while detecting none of the errors.
\nThis flashcard is a visual guide to this approach.
\nFor the kind of judge articulated in this blog post, I like to use the most powerful model I can afford in my cost/latency budget. This budget might be different than my primary model, depending on the number of examples I need to score. This can vary significantly according to the use case.
\nGuardrails are a separate but related topic. They are a way to prevent the LLM from saying/doing something harmful or inappropriate. This blog post focuses on helping you create a judge that’s aligned with business goals, especially when starting out.
\nI believe you. This blog post is not the only way to use a LLM as a judge. In fact, I’ve seen people use a LLM as a judge in all sorts of creative ways, which include ranking, classification, model selection and so-on. I’m focused on an approach that works well when you are getting started, and avoids the pitfalls of confusing metric sprawl. However, the general process of looking at the data is still central no matter what kind of judge you are building.
\nThe answer to this (and many other questions) is: do the simplest thing that works. And simple doesn’t always mean traditional ML techniques. Depending on your situation, it might be easier to use a LLM API as a classifier than to train a model and deploy it.
\nYes, potentially. I’ve only used the larger models for judges. You have to base the answer to this question on the data (i.e. the agreement with the domain expert).
\nYou have to go through the process again and measure the results.
\nYou don’t need a domain expert to grade every single example. You just need a representative sample. I don’t think you can eliminate humans completely, because the LLM still needs to be aligned to something, and that something is usually a human. As your evaluation system gets better, it naturally reduces the amount of human effort required.
\nThese are some of the resources I recommend to learn more on this topic:
\nI’m continuously learning about LLMs, and enjoy sharing my findings. If you’re interested in this journey, consider subscribing.
\nWhat to expect:
\n