--- title: 'AlignEval: Building an App to Make Evals Easy, Fun, and Automated' link: https://eugeneyan.com//writing/aligneval/ source: eugeneyan-com published: 2024-10-27T00:00:00Z updated: 2024-10-27T00:00:00Z first_seen: 2026-09-27T19:29:15.293681927Z labels: - eval - learning - llm - 🛠 - 🩷 summary: Look at and label your data, build and evaluate your LLM-evaluator, and optimize it against your labels. content: extracted html: 2024-10-27-aligneval-building-an-app-to-make-evals-easy-fun-and.html preview: file: 2024-10-27-aligneval-building-an-app-to-make-evals-easy-fun-and.preview-0270a5466836.webp width: 256 height: 134 color: '#b8b6ad' images: - source: https://eugeneyan.com/assets/og_image/aligneval-og-v4.jpg original: file: 2024-10-27-aligneval-building-an-app-to-make-evals-easy-fun-and.image-0b395cb5b753.jpg width: 1200 height: 630 color: '#d8d7d7' - source: https://eugeneyan.com/assets/upload.webp original: file: 2024-10-27-aligneval-building-an-app-to-make-evals-easy-fun-and.image-c793fe5de5cf.webp width: 1196 height: 489 variants: - file: 2024-10-27-aligneval-building-an-app-to-make-evals-easy-fun-and.image-3be9f6bb7321.webp width: 320 height: 131 color: '#f2f4f6' - source: https://eugeneyan.com/assets/table.webp original: file: 2024-10-27-aligneval-building-an-app-to-make-evals-easy-fun-and.image-4134ecde0684.webp width: 1200 height: 594 variants: - file: 2024-10-27-aligneval-building-an-app-to-make-evals-easy-fun-and.image-2015ab7ad933.webp width: 320 height: 158 color: '#e3e4e5' - source: https://eugeneyan.com/assets/labeling.webp original: file: 2024-10-27-aligneval-building-an-app-to-make-evals-easy-fun-and.image-7cad45c96f92.webp width: 1200 height: 359 color: '#fdf8c3' - source: https://eugeneyan.com/assets/evaluation.webp original: file: 2024-10-27-aligneval-building-an-app-to-make-evals-easy-fun-and.image-7c438f531f6c.webp width: 1200 height: 680 color: '#dbf9e3' - source: https://eugeneyan.com/assets/prompt.webp original: file: 2024-10-27-aligneval-building-an-app-to-make-evals-easy-fun-and.image-6896823e90ff.webp width: 1200 height: 295 color: '#fbfcfd' - source: https://eugeneyan.com/assets/metrics.webp original: file: 2024-10-27-aligneval-building-an-app-to-make-evals-easy-fun-and.image-66481e7b71e5.webp width: 891 height: 576 color: '#f2f4f6' - source: https://eugeneyan.com/assets/optimization.webp original: file: 2024-10-27-aligneval-building-an-app-to-make-evals-easy-fun-and.image-cb36ada4f260.webp width: 1200 height: 493 color: '#dae8fb' - source: https://eugeneyan.com/assets/optimize-table.webp original: file: 2024-10-27-aligneval-building-an-app-to-make-evals-easy-fun-and.image-ccd9632b1ea9.webp width: 1200 height: 407 variants: - file: 2024-10-27-aligneval-building-an-app-to-make-evals-easy-fun-and.image-d1fb651ffa4f.webp width: 320 height: 109 color: '#f7f8f8' - source: https://eugeneyan.com/assets/railway.webp original: file: 2024-10-27-aligneval-building-an-app-to-make-evals-easy-fun-and.image-30679976a041.webp width: 1200 height: 903 color: '#fafafa' - source: https://eugeneyan.com/assets/icon-twitter.svg original: file: 2024-10-27-aligneval-building-an-app-to-make-evals-easy-fun-and.image-9f73746a86e9.png width: 512 height: 512 variants: - file: 2024-10-27-aligneval-building-an-app-to-make-evals-easy-fun-and.image-10d32c0d6dac.webp width: 320 height: 320 - file: 2024-10-27-aligneval-building-an-app-to-make-evals-easy-fun-and.image-1a1653690e12.webp width: 512 height: 512 color: '#000000' - source: https://eugeneyan.com/assets/icon-linkedin.svg original: file: 2024-10-27-aligneval-building-an-app-to-make-evals-easy-fun-and.image-50dfb45d5f9e.png width: 505 height: 505 variants: - file: 2024-10-27-aligneval-building-an-app-to-make-evals-easy-fun-and.image-b7fdeee63d68.webp width: 320 height: 320 - file: 2024-10-27-aligneval-building-an-app-to-make-evals-easy-fun-and.image-003cf9435e7e.webp width: 505 height: 505 color: '#000000' - source: https://eugeneyan.com/assets/bluesky.svg original: file: 2024-10-27-aligneval-building-an-app-to-make-evals-easy-fun-and.image-cd93481613cb.png width: 600 height: 530 variants: - file: 2024-10-27-aligneval-building-an-app-to-make-evals-easy-fun-and.image-1790c4125e73.webp width: 600 height: 530 color: '#1084fd' - source: https://eugeneyan.com/assets/icon-facebook.svg original: file: 2024-10-27-aligneval-building-an-app-to-make-evals-easy-fun-and.image-68df389f82c9.png width: 256 height: 256 variants: - file: 2024-10-27-aligneval-building-an-app-to-make-evals-easy-fun-and.image-6035d4122baf.webp width: 256 height: 256 color: '#3b5998' - source: https://eugeneyan.com/assets/icon-mail.svg original: file: 2024-10-27-aligneval-building-an-app-to-make-evals-easy-fun-and.image-bf2110bd7265.png width: 512 height: 512 variants: - file: 2024-10-27-aligneval-building-an-app-to-make-evals-easy-fun-and.image-c80acd329b85.webp width: 320 height: 320 - file: 2024-10-27-aligneval-building-an-app-to-make-evals-easy-fun-and.image-84c4e78a876d.webp width: 512 height: 512 color: '#000000' - source: https://i.ytimg.com/vi/nnhMnNh_ZvU/hqdefault.jpg original: file: 2024-10-27-aligneval-building-an-app-to-make-evals-easy-fun-and.image-cede7447f011.jpg width: 480 height: 360 color: '#fcfcfc' - source: https://i.ytimg.com/vi/OJJ7aHIBtPw/hqdefault.jpg original: file: 2024-10-27-aligneval-building-an-app-to-make-evals-easy-fun-and.image-c1ee3666b054.jpg width: 480 height: 360 color: '#000000' --- Go to [aligneval.com](https://aligneval.com) to start building your own LLM-evaluator; sample data included. > **Update**: Now [integrated with LangSmith](https://blog.langchain.com/introducing-align-evals/) and [featured on VentureBeat](https://venturebeat.com/ai/langchains-align-evals-closes-the-evaluator-trust-gap-with-prompt-level-calibration)! 🎉 Every AI-powered product needs evals. But let’s face it—they’re a pain to build, hard to scale, and most teams get them wrong. As a result, many AI-powered experiences are bottlenecked on evals, sometimes delaying launches by weeks or even months. I’ve spent the past year or so wrestling with product/task-specific evals. Testing different ways to [detect](https://eugeneyan.com/writing/abstractive/) hallucinations, [finetune](https://eugeneyan.com/writing/finetuning/) evaluators, and [evaluate](https://eugeneyan.com/writing/llm-evaluators/) LLM-based evaluators. There were dead ends. There were rabbit holes. But I learned [what works and what doesn’t](https://eugeneyan.com/writing/evals/). And that’s why I’m excited to introduce [AlignEval](https://aligneval.com/), an app that makes evals easy and… fun? (Okay, I’ll settle for less painful.) It also tries to automate part of the process. AlignEval makes building LLM-evaluators as straightforward as four simple steps: - Upload a CSV file with columns for input and output. - Look at the data and label samples as pass or fail. - Define eval criteria, run the LLM-evaluator, check results. - Optimize the LLM-evaluator with dev-test splits. By the way, if you want to learn more about evals, my friends Hamel and Shreya are hosting their *final* cohort of “AI Evals for Engineers and PMs” in July. Here’s a [35% discount code](https://maven.com/parlance-labs/evals?promoCode=eugene-is-all-you-need). ## Align AI to human. Calibrate human to AI. Repeat. The key insight is that aligning AI to human preferences is only half the battle. To build effective evals, we must also **calibrate human criteria to AI output**. Many teams make the mistake of crafting elaborate eval criteria without first **looking at the data**. It’s like theorizing about user experience and defects from the ivory tower, without doing error analysis. From [Who Validates the Validators](https://arxiv.org/abs/2404.12272): “*It is impossible to completely determine evaluation criteria prior to human judging of LLM outputs.*” This leads to two types of bad criteria. First, irrelevant criteria that are a waste of time, such as generic metrics (e.g., helpfulness) or very low probability defects (e.g., grammar, spelling). Second, unrealistic, unattainable criteria that the technology isn’t ready for, such as autonomous agents back in 2023. Either way, teams squander effort that would have been better invested in evaluating actual defects that occur with moderate frequency. (I think out-of-the-box eval solutions fuel this problem, pushing generic criteria that are easy to plug and play but ignore the idiosyncrasies of your domain, product, and data.) The way to solve this—and build useful evals—is to work backward from the data. That’s what [AlignEval](https://aligneval.com/) tries to do by putting data and metrics first, not preset criteria or LLMs. ## Getting started: Upload some data **To get started, upload a CSV file containing the following columns:** - **id**: Lets you match labeled and evaluated samples to your data - **input**: The context provided to the LLM to generate the output (e.g., text to classify, news to summarize, retrieved documents for answering questions) - **output**: What the LLM generates (e.g., classification label, summary, answer) - **label**: Your judgment on whether the output passes (0) or fails (1) ![Instructions to upload a CSV file.](https://eugeneyan.com/assets/upload.webp "Instructions to upload a CSV file.") Instructions to upload a CSV file. If you don’t have a CSV handy, [download a sample](https://aligneval.com/data/fib50.csv) based on the [Factual Inconsistency Benchmark](https://huggingface.co/datasets/r-three/fib). It contains 50 news articles, each with two summaries. One is factually consistent while the other is not. Perfect for AlignEval’s binary labeling task. After uploading the CSV, you’ll see the data in a flexible table view. Adjust the width of the browser to control how wide or narrow you want the table and columns to be. ![How the table view looks after uploading a CSV file.](https://eugeneyan.com/assets/table.webp "How the table view looks after uploading a CSV file.") How the table view looks after uploading a CSV file. ## Labeling mode: Look at the data **In labeling mode, our only job is to look at the data.** AlignEval simplifies the process to a comparison between input and output fields. And for each sample, we only have to make a binary decision—pass or fail. ![Instructions for labeling mode.](https://eugeneyan.com/assets/labeling.webp "Instructions for labeling mode.") Instructions for labeling mode. Why binary labels? It leads to more accurate data, takes less time, and keeps cognitive load low. [DoorDash adopted a similar approach](https://eugeneyan.com/writing/content-moderation/#collecting-ground-truth-via-human-in-the-loop). Similarly, [Llama2](https://arxiv.org/abs/2307.09288) focused on collecting binary human preferences. The Llama2 authors also [shared at a meetup](https://www.youtube.com/live/CzR3OrOkM9w?si=gTrb6M44O941wrqb&t=913) that collecting binary preferences was much faster than writing samples for supervised finetuning. What’s important is that we look at the data with an open mind, and not write criteria based on our priors. We should also resist the urge to prematurely define criteria. We need to first immerse ourselves in the data. This reduces the risk of hunting for nonexistent defects or chasing unrealistic expectations. By understanding what the LLM actually generates, we can define more meaningful, better-calibrated criteria. [YouTube video player](https://www.youtube.com/watch?v=nnhMnNh_ZvU) Look at your data remix, courtesy of Jason Liu. Sign up for his next RAG course [here](https://maven.com/s/course/b9b1f32583). After labeling 20 rows, we unlock evaluation mode. (While evaluation mode is available after 20 labels, I think it’s too little to understand the data well enough to define criteria, and too small a sample for evaluation. I suggest aiming for 50-100 before writing criteria and running evals. The more familiar you are with your data, the better your evals will be.) ## Evaluation mode: Write criteria, evaluate the evaluator **After unlocking evaluation mode, we can now write our task-specific evaluation criteria.** Keep it simple: Evaluate on a single dimension and return either 0 (pass) or 1 (fail). ![Instructions for evaluation mode.](https://eugeneyan.com/assets/evaluation.webp "Instructions for evaluation mode.") Instructions for evaluation mode. Here’s the prompt I use for factual inconsistency classification. We start with only two sentences that define what a pass or fail looks like. We can refine it after running the LLM-evaluator and examining the explanations and predictions. ![Writing the prompt and selecting models and fields.](https://eugeneyan.com/assets/prompt.webp "Writing the prompt and selecting models and fields.") Writing the prompt and selecting models and fields. Next, we choose our model (gpt-4o-mini or claude-3-haiku) and input fields (both input and output, or output only). For most tasks, (e.g., classification, summarization, Q&A), we want to compare the output to the input. Nonetheless, for some tasks, like evaluating style guide adherence or tone of voice, we only need the output. Focusing solely on the output reduces token cost and latency while potentially improving performance. **It’s also crucial to evaluate our LLM-evaluators against the labeled data.** Hit “Evaluate” and we’ll see metrics in the top right corner: sample size, recall, precision, F1, Cohen’s $\\kappa$, and counts for true and false positives/negatives. ![Metrics to evaluate the LLM-evaluator.](https://eugeneyan.com/assets/metrics.webp "Metrics to evaluate the LLM-evaluator.") Metrics to evaluate the LLM-evaluator. If our sample size is low (e.g., 20), the metrics can fluctuate a lot, even when we rerun the same LLM-evaluator. This is due to LLM stochasticity. We can improve stability by labeling more data. Aim for at least 50 labeled samples which also unlocks optimization mode. ## Optimization mode: Semi-automated improvements 🤞 **Optimization mode is where the magic (hopefully) happens.** Click “Optimize” and let AlignEval do its thing to improve your LLM-evaluator. While it’s still in early beta, it’s achieved decent improvements for several use cases. (And no, it’s not dspy.) ![Instructions for optimization mode.](https://eugeneyan.com/assets/optimization.webp "Instructions for optimization mode.") Instructions for optimization mode. Under the hood, optimization splits the labeled data into dev and test sets. (Since we’re not training a model, we call it a development split instead of a training split.) It then runs $n$ trials on the dev set to improve F1, using the same LLM and fields from evaluation mode. After $n$ trials, the improved LLM-evaluator is evaled on the unseen test set. In the table below, F1 starts at 0.571, increases to 0.722 at trial 5 on dev, then gets F1=0.727 on test. ![Metrics from optimization trials on all, dev, and test splits.](https://eugeneyan.com/assets/optimize-table.webp "Metrics from optimization trials on all, dev, and test splits.") Metrics from optimization trials on all, dev, and test splits. Sometimes, the dev and test metrics diverge significantly. With the small sample size (25 per split, 50 in total), this likely stems from the dev split being unrepresentative of the full data distribution, including the test split. Thus, the LLM-evaluator may overfit on the dev set and fail to generalize to the test set. To improve generalization across the dev and test splits, we can label more data, ensuring that it’s diverse and representative. To be diverse is to be balanced in both 0 and 1 labels, and the variety of inputs and outputs. To be representative is to be similar to real-world examples. We can also run multiple LLM-evaluators in parallel and ensemble their scores. [PoLL](https://arxiv.org/abs/2404.18796) showed that an ensemble of three smaller LLM-evaluators outperformed gpt-4. For a visual walkthrough, here’s a demo where I look at and label 50 rows of data lol. [YouTube video player](https://www.youtube.com/watch?v=OJJ7aHIBtPw) ## Behind the scenes: How AlignEval was built As a frontend development newbie (this is my first TypeScript app), I relied on the wisdom of the crowd, polling [Twitter](https://x.com/eugeneyan/status/1828447283811402006) and [LinkedIn](https://www.linkedin.com/posts/eugeneyan_if-you-were-building-a-small-web-app-today-activity-7234224078675439617-HX75/) for advice. I also [built the same app five times](https://eugeneyan.com/writing/web-frameworks/) to get a feel for several frameworks. Specifically, I tried FastHTML, Next.js, SvelteKit, and FastAPI + HTML (my go-to for prototyping). From early prototypes, Next.js seemed to be a good fit. Intuitive, scalable, and the perfect excuse to learn TypeScript. And thanks to Cursor, my beginner questions were answered easily: What’s the difference between `const` and `let`? What is a `prop`? It also made it easier to build UX components and fix bugs 2-3x faster, keeping velocity and motivation high. For the backend, Python + FastAPI was the obvious choice. Python’s ecosystem of data analysis and machine learning made it easier to experiment with optimization mode. When it came to LLMs, I went with the smallest models from the biggest labs: gpt-4o-mini and claude-3-haiku (soon to be claude-3.5-haiku). They’re cheap, fast, and good enough for binary classification LLM-evaluators. It’s the perfect combo for a free app. And with each release, these small models just keep getting more and more capable. For gpt-4o-mini, I used the [structured output](https://platform.openai.com/docs/guides/structured-outputs) functionality that’s still in beta. It allows users to define the desired output schema using Zod (JavaScript) or Pydantic (Python). Here’s an example in Javascript and its equivalent in Python: ```javascript import { z } from "zod"; const EvaluationResponse = z.object({ explanation: z.string(), prediction: z.string().refine((val) => val === '0' || val === '1') }); ``` ```python from pydantic import BaseModel class EvaluationResponse(BaseModel): explanation: str prediction: str ``` For claude-3-haiku, I [defined the XML output](https://eugeneyan.com/writing/prompting/#structured-input-and-output) via the prompt. From what I’ve seen on the job and in AlignEval, this works 99.9% of the time. Here’s the prompt I used for AlignEval. ```plaintext Evaluate the output based on the provided criteria. First, in the provided, think through your evaluation step by step. Then, provide a binary prediction (0 or 1) within . ``` To decide on hosting, I conducted a [Twitter poll](https://x.com/eugeneyan/status/1828450401311764741) and consulted documentation for options like Dokku, Coolify, Vercel, etc. [Hamel’s write-up on Dokku](https://hamel.dev/blog/posts/dokku/) was particularly helpful. While I initially considered hosting the app on Hetzner with Dokku (another opportunity to learn something new), I ultimately decided to take the lazy path (read: prioritize ruthlessly) and start with hosting on Railway. It was a good decision. Hosting on Railway has been a breeze. After setting up the project via their web UI and linking it via the CLI (`railway link`), pushing new updates is as simple as `railway up`. Each update takes one to two minutes. Railway also comes with several database options, including Postgres and Redis. While a VPS might be cheaper in the long run, at this stage, [YAGNI](https://en.wikipedia.org/wiki/You_aren%27t_gonna_need_it). Fiddling with a VPS would have been a distraction from improving the UX, implementing features, and figuring out optimization mode. ![My simple Railway setup.](https://eugeneyan.com/assets/railway.webp "My simple Railway setup.") My simple Railway setup > If you’d like to get started with Railway, please use my [affiliate code](https://railway.app?referralCode=fMY11k)—it’ll help with hosting costs and keep AlignEval running! Going from a laptop prototype to a public beta took some effort. The work included: - Splitting into Next.js frontend, FastAPI backend, migrating from SQLite to Postgres - Implementing real-time feedback for labeling, evaluation, and optimization - Adding features to simplify labeling (pass/fail buttons), customize models and fields (dropdowns, integration with model providers), and more - Improving the UI (as rough as it looks now, it was way worse before lol) - Squashing lots of bugs (and probably missing many more) Here’s the to-do list I had; there were many more tasks and bugs that I neglected to track. Because I timeboxed the effort to October, a few items were not completed. But if AlignEval gains traction and proves useful, I might revisit the list. ```markdown ## Todos - [x] Split deployment into frontend and backend - https://help.railway.app/questions/how-to-expose-a-fast-api-backend-service-a1712631 - https://help.railway.app/questions/econnrefused-when-calling-another-servic-5662d969#p-1 - ~~[ ] Or consider something like this: https://vercel.com/templates/next.js/nextjs-fastapi-starter~~ (only works with Vercel) - [x] Add external database (Railway Postgres) - [x] Write interface to ensure compatibility between SQLite and Postgres - [x] Connect label-app to Railway Postgres - [x] Connect label-hpo to Railway Postgres - [x] Show feedback to users while optimization is running - [x] Update table names to use CSV file name - [x] Add support for multiple tables - [x] Replace tablename parameter with filename parameter - [x] Uploading a CSV file with an existing name should return the existing table - [x] Add llm-evaluator optimization to label-hpo - [x] Add dropdown to select evaluation fields - [x] Add ability to delete optimization table, with password - [x] Add progress bar to gamify labeling, evaluation and optimization - [x] Add buttons to click pass and fail - [x] Add usage count (upload (files, rows), evaluate (files, rows), optimize (files, trials)) - [x] Add info button intro next to header to explain the site - [x] Update favicon - [x] Add support for OpenAI models - [x] Add balanced val-test split for optimization - [x] Add columns to optimization metrics: split (dev, test), model, evaluation_fields - [x] Set max upload rows to env variable - [ ] Fix bug where alert pops up: "Failed to handle optimization: Unexpected token '<', " Yan, Ziyou. (Oct 2024). AlignEval: Building an App to Make Evals Easy, Fun, and Automated. eugeneyan.com. https://eugeneyan.com/writing/aligneval/. or ``` @article{yan2024aligneval, title = {AlignEval: Building an App to Make Evals Easy, Fun, and Automated}, author = {Yan, Ziyou}, journal = {eugeneyan.com}, year = {2024}, month = {Oct}, url = {https://eugeneyan.com/writing/aligneval/} } ``` Share on: ![](https://eugeneyan.com/assets/icon-twitter.svg) ![](https://eugeneyan.com/assets/icon-linkedin.svg) ![](https://eugeneyan.com/assets/bluesky.svg) ![](https://eugeneyan.com/assets/icon-facebook.svg) ![](https://eugeneyan.com/assets/icon-mail.svg) Join **11,800+** readers getting updates on machine learning, RecSys, LLMs, and engineering.