{ "@context": "https://schema.org", "@id": "https://aggr.aymericbeaumet.com/items/eugeneyan-com/2024-10-27-aligneval-building-an-app-to-make-evals-easy-fun-and/#webpage", "@type": [ "WebPage", "ArchiveComponent" ], "articleBody": "Go to [aligneval.com](https://aligneval.com) to start building your own LLM-evaluator; sample data included.\n\n> **Update**: Now [integrated with LangSmith](https://blog.langchain.com/introducing-align-evals/) and [featured on VentureBeat](https://venturebeat.com/ai/langchains-align-evals-closes-the-evaluator-trust-gap-with-prompt-level-calibration)! 🎉\n\nEvery AI-powered product needs evals. But let’s face it—they’re a pain to build, hard to scale, and most teams get them wrong. As a result, many AI-powered experiences are bottlenecked on evals, sometimes delaying launches by weeks or even months.\n\nI’ve spent the past year or so wrestling with product/task-specific evals. Testing different ways to [detect](https://eugeneyan.com/writing/abstractive/) hallucinations, [finetune](https://eugeneyan.com/writing/finetuning/) evaluators, and [evaluate](https://eugeneyan.com/writing/llm-evaluators/) LLM-based evaluators. There were dead ends. There were rabbit holes. But I learned [what works and what doesn’t](https://eugeneyan.com/writing/evals/).\n\nAnd that’s why I’m excited to introduce [AlignEval](https://aligneval.com/), an app that makes evals easy and… fun? (Okay, I’ll settle for less painful.) It also tries to automate part of the process. AlignEval makes building LLM-evaluators as straightforward as four simple steps:\n\n- Upload a CSV file with columns for input and output.\n- Look at the data and label samples as pass or fail.\n- Define eval criteria, run the LLM-evaluator, check results.\n- Optimize the LLM-evaluator with dev-test splits.\n\nBy the way, if you want to learn more about evals, my friends Hamel and Shreya are hosting their *final* cohort of “AI Evals for Engineers and PMs” in July. Here’s a [35% discount code](https://maven.com/parlance-labs/evals?promoCode=eugene-is-all-you-need).\n\n## Align AI to human. Calibrate human to AI. Repeat.\n\nThe key insight is that aligning AI to human preferences is only half the battle. To build effective evals, we must also **calibrate human criteria to AI output**.\n\nMany teams make the mistake of crafting elaborate eval criteria without first **looking at the data**. It’s like theorizing about user experience and defects from the ivory tower, without doing error analysis. From [Who Validates the Validators](https://arxiv.org/abs/2404.12272): “*It is impossible to completely determine evaluation criteria prior to human judging of LLM outputs.*”\n\nThis leads to two types of bad criteria. First, irrelevant criteria that are a waste of time, such as generic metrics (e.g., helpfulness) or very low probability defects (e.g., grammar, spelling). Second, unrealistic, unattainable criteria that the technology isn’t ready for, such as autonomous agents back in 2023. Either way, teams squander effort that would have been better invested in evaluating actual defects that occur with moderate frequency.\n\n(I think out-of-the-box eval solutions fuel this problem, pushing generic criteria that are easy to plug and play but ignore the idiosyncrasies of your domain, product, and data.)\n\nThe way to solve this—and build useful evals—is to work backward from the data. That’s what [AlignEval](https://aligneval.com/) tries to do by putting data and metrics first, not preset criteria or LLMs.\n\n## Getting started: Upload some data\n\n**To get started, upload a CSV file containing the following columns:**\n\n- **id**: Lets you match labeled and evaluated samples to your data\n- **input**: The context provided to the LLM to generate the output (e.g., text to classify, news to summarize, retrieved documents for answering questions)\n- **output**: What the LLM generates (e.g., classification label, summary, answer)\n- **label**: Your judgment on whether the output passes (0) or fails (1)\n\n![Instructions to upload a CSV file.](https://eugeneyan.com/assets/upload.webp \"Instructions to upload a CSV file.\")\n\nInstructions to upload a CSV file.\n\nIf you don’t have a CSV handy, [download a sample](https://aligneval.com/data/fib50.csv) based on the [Factual Inconsistency Benchmark](https://huggingface.co/datasets/r-three/fib). It contains 50 news articles, each with two summaries. One is factually consistent while the other is not. Perfect for AlignEval’s binary labeling task.\n\nAfter uploading the CSV, you’ll see the data in a flexible table view. Adjust the width of the browser to control how wide or narrow you want the table and columns to be.\n\n![How the table view looks after uploading a CSV file.](https://eugeneyan.com/assets/table.webp \"How the table view looks after uploading a CSV file.\")\n\nHow the table view looks after uploading a CSV file.\n\n## Labeling mode: Look at the data\n\n**In labeling mode, our only job is to look at the data.** AlignEval simplifies the process to a comparison between input and output fields. And for each sample, we only have to make a binary decision—pass or fail.\n\n![Instructions for labeling mode.](https://eugeneyan.com/assets/labeling.webp \"Instructions for labeling mode.\")\n\nInstructions for labeling mode.\n\nWhy binary labels? It leads to more accurate data, takes less time, and keeps cognitive load low. [DoorDash adopted a similar approach](https://eugeneyan.com/writing/content-moderation/#collecting-ground-truth-via-human-in-the-loop). Similarly, [Llama2](https://arxiv.org/abs/2307.09288) focused on collecting binary human preferences. The Llama2 authors also [shared at a meetup](https://www.youtube.com/live/CzR3OrOkM9w?si=gTrb6M44O941wrqb&t=913) that collecting binary preferences was much faster than writing samples for supervised finetuning.\n\nWhat’s important is that we look at the data with an open mind, and not write criteria based on our priors. We should also resist the urge to prematurely define criteria. We need to first immerse ourselves in the data. This reduces the risk of hunting for nonexistent defects or chasing unrealistic expectations. By understanding what the LLM actually generates, we can define more meaningful, better-calibrated criteria.\n\n[YouTube video player](https://www.youtube.com/watch?v=nnhMnNh_ZvU)\n\nLook at your data remix, courtesy of Jason Liu. Sign up for his next RAG course [here](https://maven.com/s/course/b9b1f32583).\n\nAfter labeling 20 rows, we unlock evaluation mode. (While evaluation mode is available after 20 labels, I think it’s too little to understand the data well enough to define criteria, and too small a sample for evaluation. I suggest aiming for 50-100 before writing criteria and running evals. The more familiar you are with your data, the better your evals will be.)\n\n## Evaluation mode: Write criteria, evaluate the evaluator\n\n**After unlocking evaluation mode, we can now write our task-specific evaluation criteria.** Keep it simple: Evaluate on a single dimension and return either 0 (pass) or 1 (fail).\n\n![Instructions for evaluation mode.](https://eugeneyan.com/assets/evaluation.webp \"Instructions for evaluation mode.\")\n\nInstructions for evaluation mode.\n\nHere’s the prompt I use for factual inconsistency classification. We start with only two sentences that define what a pass or fail looks like. We can refine it after running the LLM-evaluator and examining the explanations and predictions.\n\n![Writing the prompt and selecting models and fields.](https://eugeneyan.com/assets/prompt.webp \"Writing the prompt and selecting models and fields.\")\n\nWriting the prompt and selecting models and fields.\n\nNext, we choose our model (gpt-4o-mini or claude-3-haiku) and input fields (both input and output, or output only). For most tasks, (e.g., classification, summarization, Q&A), we want to compare the output to the input. Nonetheless, for some tasks, like evaluating style guide adherence or tone of voice, we only need the output. Focusing solely on the output reduces token cost and latency while potentially improving performance.\n\n**It’s also crucial to evaluate our LLM-evaluators against the labeled data.** Hit “Evaluate” and we’ll see metrics in the top right corner: sample size, recall, precision, F1, Cohen’s $\\\\kappa$, and counts for true and false positives/negatives.\n\n![Metrics to evaluate the LLM-evaluator.](https://eugeneyan.com/assets/metrics.webp \"Metrics to evaluate the LLM-evaluator.\")\n\nMetrics to evaluate the LLM-evaluator.\n\nIf our sample size is low (e.g., 20), the metrics can fluctuate a lot, even when we rerun the same LLM-evaluator. This is due to LLM stochasticity. We can improve stability by labeling more data. Aim for at least 50 labeled samples which also unlocks optimization mode.\n\n## Optimization mode: Semi-automated improvements 🤞\n\n**Optimization mode is where the magic (hopefully) happens.** Click “Optimize” and let AlignEval do its thing to improve your LLM-evaluator. While it’s still in early beta, it’s achieved decent improvements for several use cases. (And no, it’s not dspy.)\n\n![Instructions for optimization mode.](https://eugeneyan.com/assets/optimization.webp \"Instructions for optimization mode.\")\n\nInstructions for optimization mode.\n\nUnder the hood, optimization splits the labeled data into dev and test sets. (Since we’re not training a model, we call it a development split instead of a training split.) It then runs $n$ trials on the dev set to improve F1, using the same LLM and fields from evaluation mode. After $n$ trials, the improved LLM-evaluator is evaled on the unseen test set. In the table below, F1 starts at 0.571, increases to 0.722 at trial 5 on dev, then gets F1=0.727 on test.\n\n![Metrics from optimization trials on all, dev, and test splits.](https://eugeneyan.com/assets/optimize-table.webp \"Metrics from optimization trials on all, dev, and test splits.\")\n\nMetrics from optimization trials on all, dev, and test splits.\n\nSometimes, the dev and test metrics diverge significantly. With the small sample size (25 per split, 50 in total), this likely stems from the dev split being unrepresentative of the full data distribution, including the test split. Thus, the LLM-evaluator may overfit on the dev set and fail to generalize to the test set.\n\nTo improve generalization across the dev and test splits, we can label more data, ensuring that it’s diverse and representative. To be diverse is to be balanced in both 0 and 1 labels, and the variety of inputs and outputs. To be representative is to be similar to real-world examples. We can also run multiple LLM-evaluators in parallel and ensemble their scores. [PoLL](https://arxiv.org/abs/2404.18796) showed that an ensemble of three smaller LLM-evaluators outperformed gpt-4.\n\nFor a visual walkthrough, here’s a demo where I look at and label 50 rows of data lol.\n\n[YouTube video player](https://www.youtube.com/watch?v=OJJ7aHIBtPw)\n\n## Behind the scenes: How AlignEval was built\n\nAs a frontend development newbie (this is my first TypeScript app), I relied on the wisdom of the crowd, polling [Twitter](https://x.com/eugeneyan/status/1828447283811402006) and [LinkedIn](https://www.linkedin.com/posts/eugeneyan_if-you-were-building-a-small-web-app-today-activity-7234224078675439617-HX75/) for advice. I also [built the same app five times](https://eugeneyan.com/writing/web-frameworks/) to get a feel for several frameworks. Specifically, I tried FastHTML, Next.js, SvelteKit, and FastAPI + HTML (my go-to for prototyping).\n\nFrom early prototypes, Next.js seemed to be a good fit. Intuitive, scalable, and the perfect excuse to learn TypeScript. And thanks to Cursor, my beginner questions were answered easily: What’s the difference between `const` and `let`? What is a `prop`? It also made it easier to build UX components and fix bugs 2-3x faster, keeping velocity and motivation high.\n\nFor the backend, Python + FastAPI was the obvious choice. Python’s ecosystem of data analysis and machine learning made it easier to experiment with optimization mode.\n\nWhen it came to LLMs, I went with the smallest models from the biggest labs: gpt-4o-mini and claude-3-haiku (soon to be claude-3.5-haiku). They’re cheap, fast, and good enough for binary classification LLM-evaluators. It’s the perfect combo for a free app. And with each release, these small models just keep getting more and more capable.\n\nFor gpt-4o-mini, I used the [structured output](https://platform.openai.com/docs/guides/structured-outputs) functionality that’s still in beta. It allows users to define the desired output schema using Zod (JavaScript) or Pydantic (Python). Here’s an example in Javascript and its equivalent in Python:\n\n```javascript\nimport { z } from \"zod\";\n\nconst EvaluationResponse = z.object({\n explanation: z.string(),\n prediction: z.string().refine((val) => val === '0' || val === '1')\n});\n```\n\n```python\nfrom pydantic import BaseModel\n\nclass EvaluationResponse(BaseModel):\n explanation: str\n prediction: str\n```\n\nFor claude-3-haiku, I [defined the XML output](https://eugeneyan.com/writing/prompting/#structured-input-and-output) via the prompt. From what I’ve seen on the job and in AlignEval, this works 99.9% of the time. Here’s the prompt I used for AlignEval.\n\n```plaintext\nEvaluate the output based on the provided criteria. First, in the provided, think through your evaluation step by step.\n\nThen, provide a binary prediction (0 or 1) within .\n```\n\nTo decide on hosting, I conducted a [Twitter poll](https://x.com/eugeneyan/status/1828450401311764741) and consulted documentation for options like Dokku, Coolify, Vercel, etc. [Hamel’s write-up on Dokku](https://hamel.dev/blog/posts/dokku/) was particularly helpful. While I initially considered hosting the app on Hetzner with Dokku (another opportunity to learn something new), I ultimately decided to take the lazy path (read: prioritize ruthlessly) and start with hosting on Railway. It was a good decision.\n\nHosting on Railway has been a breeze. After setting up the project via their web UI and linking it via the CLI (`railway link`), pushing new updates is as simple as `railway up`. Each update takes one to two minutes. Railway also comes with several database options, including Postgres and Redis. While a VPS might be cheaper in the long run, at this stage, [YAGNI](https://en.wikipedia.org/wiki/You_aren%27t_gonna_need_it). Fiddling with a VPS would have been a distraction from improving the UX, implementing features, and figuring out optimization mode.\n\n![My simple Railway setup.](https://eugeneyan.com/assets/railway.webp \"My simple Railway setup.\")\n\nMy simple Railway setup\n\n> If you’d like to get started with Railway, please use my [affiliate code](https://railway.app?referralCode=fMY11k)—it’ll help with hosting costs and keep AlignEval running!\n\nGoing from a laptop prototype to a public beta took some effort. The work included:\n\n- Splitting into Next.js frontend, FastAPI backend, migrating from SQLite to Postgres\n- Implementing real-time feedback for labeling, evaluation, and optimization\n- Adding features to simplify labeling (pass/fail buttons), customize models and fields (dropdowns, integration with model providers), and more\n- Improving the UI (as rough as it looks now, it was way worse before lol)\n- Squashing lots of bugs (and probably missing many more)\n\nHere’s the to-do list I had; there were many more tasks and bugs that I neglected to track. Because I timeboxed the effort to October, a few items were not completed. But if AlignEval gains traction and proves useful, I might revisit the list.\n\n```markdown\n## Todos\n- [x] Split deployment into frontend and backend\n - https://help.railway.app/questions/how-to-expose-a-fast-api-backend-service-a1712631\n - https://help.railway.app/questions/econnrefused-when-calling-another-servic-5662d969#p-1\n - ~~[ ] Or consider something like this: https://vercel.com/templates/next.js/nextjs-fastapi-starter~~ (only works with Vercel)\n- [x] Add external database (Railway Postgres)\n- [x] Write interface to ensure compatibility between SQLite and Postgres\n- [x] Connect label-app to Railway Postgres\n- [x] Connect label-hpo to Railway Postgres\n- [x] Show feedback to users while optimization is running\n- [x] Update table names to use CSV file name\n- [x] Add support for multiple tables\n- [x] Replace tablename parameter with filename parameter\n- [x] Uploading a CSV file with an existing name should return the existing table\n- [x] Add llm-evaluator optimization to label-hpo\n- [x] Add dropdown to select evaluation fields\n- [x] Add ability to delete optimization table, with password\n- [x] Add progress bar to gamify labeling, evaluation and optimization\n- [x] Add buttons to click pass and fail\n- [x] Add usage count (upload (files, rows), evaluate (files, rows), optimize (files, trials))\n- [x] Add info button intro next to header to explain the site\n- [x] Update favicon\n- [x] Add support for OpenAI models\n- [x] Add balanced val-test split for optimization\n- [x] Add columns to optimization metrics: split (dev, test), model, evaluation_fields\n- [x] Set max upload rows to env variable\n- [ ] Fix bug where alert pops up: \"Failed to handle optimization: Unexpected token '<', \" Yan, Ziyou. (Oct 2024). AlignEval: Building an App to Make Evals Easy, Fun, and Automated. eugeneyan.com. https://eugeneyan.com/writing/aligneval/.\n\nor\n\n```\n@article{yan2024aligneval,\n title = {AlignEval: Building an App to Make Evals Easy, Fun, and Automated},\n author = {Yan, Ziyou},\n journal = {eugeneyan.com},\n year = {2024},\n month = {Oct},\n url = {https://eugeneyan.com/writing/aligneval/}\n}\n```\n\nShare on:\n\n![](https://eugeneyan.com/assets/icon-twitter.svg)\n\n![](https://eugeneyan.com/assets/icon-linkedin.svg)\n\n![](https://eugeneyan.com/assets/bluesky.svg)\n\n![](https://eugeneyan.com/assets/icon-facebook.svg)\n\n![](https://eugeneyan.com/assets/icon-mail.svg)\n\nJoin **11,800+** readers getting updates on machine learning, RecSys, LLMs, and engineering.\n", "authors": [], "captured_at": "2026-09-27T19:29:15.293681927+00:00", "category": "programming", "content_html": "

Go to aligneval.com to start building your own LLM-evaluator; sample data included.

\n
\n

Update: Now integrated with LangSmith and featured on VentureBeat! 🎉

\n
\n

Every AI-powered product needs evals. But let’s face it—they’re a pain to build, hard to scale, and most teams get them wrong. As a result, many AI-powered experiences are bottlenecked on evals, sometimes delaying launches by weeks or even months.

\n

I’ve spent the past year or so wrestling with product/task-specific evals. Testing different ways to detect hallucinations, finetune evaluators, and evaluate LLM-based evaluators. There were dead ends. There were rabbit holes. But I learned what works and what doesn’t.

\n

And that’s why I’m excited to introduce AlignEval, an app that makes evals easy and… fun? (Okay, I’ll settle for less painful.) It also tries to automate part of the process. AlignEval makes building LLM-evaluators as straightforward as four simple steps:

\n\n

By the way, if you want to learn more about evals, my friends Hamel and Shreya are hosting their final cohort of “AI Evals for Engineers and PMs” in July. Here’s a 35% discount code.

\n

Align AI to human. Calibrate human to AI. Repeat.

\n

The key insight is that aligning AI to human preferences is only half the battle. To build effective evals, we must also calibrate human criteria to AI output.

\n

Many teams make the mistake of crafting elaborate eval criteria without first looking at the data. It’s like theorizing about user experience and defects from the ivory tower, without doing error analysis. From Who Validates the Validators: “It is impossible to completely determine evaluation criteria prior to human judging of LLM outputs.”

\n

This leads to two types of bad criteria. First, irrelevant criteria that are a waste of time, such as generic metrics (e.g., helpfulness) or very low probability defects (e.g., grammar, spelling). Second, unrealistic, unattainable criteria that the technology isn’t ready for, such as autonomous agents back in 2023. Either way, teams squander effort that would have been better invested in evaluating actual defects that occur with moderate frequency.

\n

(I think out-of-the-box eval solutions fuel this problem, pushing generic criteria that are easy to plug and play but ignore the idiosyncrasies of your domain, product, and data.)

\n

The way to solve this—and build useful evals—is to work backward from the data. That’s what AlignEval tries to do by putting data and metrics first, not preset criteria or LLMs.

\n

Getting started: Upload some data

\n

To get started, upload a CSV file containing the following columns:

\n\n

\"Instructions

\n

Instructions to upload a CSV file.

\n

If you don’t have a CSV handy, download a sample based on the Factual Inconsistency Benchmark. It contains 50 news articles, each with two summaries. One is factually consistent while the other is not. Perfect for AlignEval’s binary labeling task.

\n

After uploading the CSV, you’ll see the data in a flexible table view. Adjust the width of the browser to control how wide or narrow you want the table and columns to be.

\n

\"How

\n

How the table view looks after uploading a CSV file.

\n

Labeling mode: Look at the data

\n

In labeling mode, our only job is to look at the data. AlignEval simplifies the process to a comparison between input and output fields. And for each sample, we only have to make a binary decision—pass or fail.

\n

\"Instructions

\n

Instructions for labeling mode.

\n

Why binary labels? It leads to more accurate data, takes less time, and keeps cognitive load low. DoorDash adopted a similar approach. Similarly, Llama2 focused on collecting binary human preferences. The Llama2 authors also shared at a meetup that collecting binary preferences was much faster than writing samples for supervised finetuning.

\n

What’s important is that we look at the data with an open mind, and not write criteria based on our priors. We should also resist the urge to prematurely define criteria. We need to first immerse ourselves in the data. This reduces the risk of hunting for nonexistent defects or chasing unrealistic expectations. By understanding what the LLM actually generates, we can define more meaningful, better-calibrated criteria.

\n

YouTube video player

\n

Look at your data remix, courtesy of Jason Liu. Sign up for his next RAG course here.

\n

After labeling 20 rows, we unlock evaluation mode. (While evaluation mode is available after 20 labels, I think it’s too little to understand the data well enough to define criteria, and too small a sample for evaluation. I suggest aiming for 50-100 before writing criteria and running evals. The more familiar you are with your data, the better your evals will be.)

\n

Evaluation mode: Write criteria, evaluate the evaluator

\n

After unlocking evaluation mode, we can now write our task-specific evaluation criteria. Keep it simple: Evaluate on a single dimension and return either 0 (pass) or 1 (fail).

\n

\"Instructions

\n

Instructions for evaluation mode.

\n

Here’s the prompt I use for factual inconsistency classification. We start with only two sentences that define what a pass or fail looks like. We can refine it after running the LLM-evaluator and examining the explanations and predictions.

\n

\"Writing

\n

Writing the prompt and selecting models and fields.

\n

Next, we choose our model (gpt-4o-mini or claude-3-haiku) and input fields (both input and output, or output only). For most tasks, (e.g., classification, summarization, Q&A), we want to compare the output to the input. Nonetheless, for some tasks, like evaluating style guide adherence or tone of voice, we only need the output. Focusing solely on the output reduces token cost and latency while potentially improving performance.

\n

It’s also crucial to evaluate our LLM-evaluators against the labeled data. Hit “Evaluate” and we’ll see metrics in the top right corner: sample size, recall, precision, F1, Cohen’s κ, and counts for true and false positives/negatives.

\n

\"Metrics

\n

Metrics to evaluate the LLM-evaluator.

\n

If our sample size is low (e.g., 20), the metrics can fluctuate a lot, even when we rerun the same LLM-evaluator. This is due to LLM stochasticity. We can improve stability by labeling more data. Aim for at least 50 labeled samples which also unlocks optimization mode.

\n

Optimization mode: Semi-automated improvements 🤞

\n

Optimization mode is where the magic (hopefully) happens. Click “Optimize” and let AlignEval do its thing to improve your LLM-evaluator. While it’s still in early beta, it’s achieved decent improvements for several use cases. (And no, it’s not dspy.)

\n

\"Instructions

\n

Instructions for optimization mode.

\n

Under the hood, optimization splits the labeled data into dev and test sets. (Since we’re not training a model, we call it a development split instead of a training split.) It then runs n trials on the dev set to improve F1, using the same LLM and fields from evaluation mode. After n trials, the improved LLM-evaluator is evaled on the unseen test set. In the table below, F1 starts at 0.571, increases to 0.722 at trial 5 on dev, then gets F1=0.727 on test.

\n

\"Metrics

\n

Metrics from optimization trials on all, dev, and test splits.

\n

Sometimes, the dev and test metrics diverge significantly. With the small sample size (25 per split, 50 in total), this likely stems from the dev split being unrepresentative of the full data distribution, including the test split. Thus, the LLM-evaluator may overfit on the dev set and fail to generalize to the test set.

\n

To improve generalization across the dev and test splits, we can label more data, ensuring that it’s diverse and representative. To be diverse is to be balanced in both 0 and 1 labels, and the variety of inputs and outputs. To be representative is to be similar to real-world examples. We can also run multiple LLM-evaluators in parallel and ensemble their scores. PoLL showed that an ensemble of three smaller LLM-evaluators outperformed gpt-4.

\n

For a visual walkthrough, here’s a demo where I look at and label 50 rows of data lol.

\n

YouTube video player

\n

Behind the scenes: How AlignEval was built

\n

As a frontend development newbie (this is my first TypeScript app), I relied on the wisdom of the crowd, polling Twitter and LinkedIn for advice. I also built the same app five times to get a feel for several frameworks. Specifically, I tried FastHTML, Next.js, SvelteKit, and FastAPI + HTML (my go-to for prototyping).

\n

From early prototypes, Next.js seemed to be a good fit. Intuitive, scalable, and the perfect excuse to learn TypeScript. And thanks to Cursor, my beginner questions were answered easily: What’s the difference between const and let? What is a prop? It also made it easier to build UX components and fix bugs 2-3x faster, keeping velocity and motivation high.

\n

For the backend, Python + FastAPI was the obvious choice. Python’s ecosystem of data analysis and machine learning made it easier to experiment with optimization mode.

\n

When it came to LLMs, I went with the smallest models from the biggest labs: gpt-4o-mini and claude-3-haiku (soon to be claude-3.5-haiku). They’re cheap, fast, and good enough for binary classification LLM-evaluators. It’s the perfect combo for a free app. And with each release, these small models just keep getting more and more capable.

\n

For gpt-4o-mini, I used the structured output functionality that’s still in beta. It allows users to define the desired output schema using Zod (JavaScript) or Pydantic (Python). Here’s an example in Javascript and its equivalent in Python:

\n
import { z } from "zod";\n\nconst EvaluationResponse = z.object({\n  explanation: z.string(),\n  prediction: z.string().refine((val) => val === '0' || val === '1')\n});\n
\n
from pydantic import BaseModel\n\nclass EvaluationResponse(BaseModel):\n  explanation: str\n  prediction: str\n
\n

For claude-3-haiku, I defined the XML output via the prompt. From what I’ve seen on the job and in AlignEval, this works 99.9% of the time. Here’s the prompt I used for AlignEval.

\n
Evaluate the output based on the provided criteria. First, in the <sketchpad> provided, think through your evaluation step by step.\n\nThen, provide a binary prediction (0 or 1) within <prediction>.\n
\n

To decide on hosting, I conducted a Twitter poll and consulted documentation for options like Dokku, Coolify, Vercel, etc. Hamel’s write-up on Dokku was particularly helpful. While I initially considered hosting the app on Hetzner with Dokku (another opportunity to learn something new), I ultimately decided to take the lazy path (read: prioritize ruthlessly) and start with hosting on Railway. It was a good decision.

\n

Hosting on Railway has been a breeze. After setting up the project via their web UI and linking it via the CLI (railway link), pushing new updates is as simple as railway up. Each update takes one to two minutes. Railway also comes with several database options, including Postgres and Redis. While a VPS might be cheaper in the long run, at this stage, YAGNI. Fiddling with a VPS would have been a distraction from improving the UX, implementing features, and figuring out optimization mode.

\n

\"My

\n

My simple Railway setup

\n
\n

If you’d like to get started with Railway, please use my affiliate code—it’ll help with hosting costs and keep AlignEval running!

\n
\n

Going from a laptop prototype to a public beta took some effort. The work included:

\n\n

Here’s the to-do list I had; there were many more tasks and bugs that I neglected to track. Because I timeboxed the effort to October, a few items were not completed. But if AlignEval gains traction and proves useful, I might revisit the list.

\n
## Todos\n- [x] Split deployment into frontend and backend\n  - https://help.railway.app/questions/how-to-expose-a-fast-api-backend-service-a1712631\n  - https://help.railway.app/questions/econnrefused-when-calling-another-servic-5662d969#p-1\n  - ~~[ ] Or consider something like this: https://vercel.com/templates/next.js/nextjs-fastapi-starter~~ (only works with Vercel)\n- [x] Add external database (Railway Postgres)\n- [x] Write interface to ensure compatibility between SQLite and Postgres\n- [x] Connect label-app to Railway Postgres\n- [x] Connect label-hpo to Railway Postgres\n- [x] Show feedback to users while optimization is running\n- [x] Update table names to use CSV file name\n- [x] Add support for multiple tables\n- [x] Replace tablename parameter with filename parameter\n- [x] Uploading a CSV file with an existing name should return the existing table\n- [x] Add llm-evaluator optimization to label-hpo\n- [x] Add dropdown to select evaluation fields\n- [x] Add ability to delete optimization table, with password\n- [x] Add progress bar to gamify labeling, evaluation and optimization\n- [x] Add buttons to click pass and fail\n- [x] Add usage count (upload (files, rows), evaluate (files, rows), optimize (files, trials))\n- [x] Add info button intro next to header to explain the site\n- [x] Update favicon\n- [x] Add support for OpenAI models\n- [x] Add balanced val-test split for optimization\n- [x] Add columns to optimization metrics: split (dev, test), model, evaluation_fields\n- [x] Set max upload rows to env variable\n- [ ] Fix bug where alert pops up: "Failed to handle optimization: Unexpected token '<', "<!DOCTYPE "... is not valid JSON"\n  - Still pops up every now and then\n- [x] Fix bug where evaluation alert pops up even when optimization is unlocked\n- [x] Add popup to see full prompt\n- [x] Add button to copy prompt\n- [x] Add completion bar to optimization sticky\n- [x] Make site look retro and like a game\n- [x] Prompt users to rename sample data before downloading\n- [x] Update delete optimization to reset optimization, and set password to Twitter handle\n- [x] Add "don't show me this again" checkbox\n- [ ] Improve how we poll for data\n- [ ] Add button to donate with Stripe payment\n- [ ] Add button to optimize more with Stripe payment\n\n## Todos (OE)\n- [x] Separate data definitions to definitions.ts\n- [x] Separate fetch data queries to data.ts\n- [x] Separate data mutations to actions.ts\n- [x] Separate components into component.tsx under ui which are imported in page.tsx\n
\n

Building AlignEval has been a ton of fun, and writing about the journey was a great way to reflect. It’s easily my favorite project of the year, because I got to build an app and learn a lot along the way. (That said, there’s still November and December to try to beat this.)

\n

If you’re tired of spending days or weeks on evals and aligning LLM-evaluators, try AlignEval. It’s designed to streamline your workflow and focus on what matters most:

\n\n

Here’s the code in case you want to deploy your own instance. It’s my first TypeScript project, so feedback welcome! How could I have organized it better? What best practices should I have followed? And beyond the code, what have you found useful when building LLM-evaluators? Comment below or DM me.

\n

Thanks to the incredible folks who made discussing this topic so much fun, and provided helpful feedback at early demos: Shreya Shankar, Hamel Husain, Kyle Corbitt, David Corbitt, Saumya Gandhi, Swyx, Eugene Cheah, and Dennis Taylor. And thanks to Hamel Husain for reading early drafts of this piece.

\n

By the way, if you want to learn more about evals, my friends Hamel and Shreya are hosting their final cohort of “AI Evals for Engineers and PMs” in July. Here’s a 35% discount code.

\n

Further reading

\n\n

If you found this useful, please cite this write-up as:

\n
\n

Yan, Ziyou. (Oct 2024). AlignEval: Building an App to Make Evals Easy, Fun, and Automated. eugeneyan.com. https://eugeneyan.com/writing/aligneval/.

\n
\n

or

\n
@article{yan2024aligneval,\n  title   = {AlignEval: Building an App to Make Evals Easy, Fun, and Automated},\n  author  = {Yan, Ziyou},\n  journal = {eugeneyan.com},\n  year    = {2024},\n  month   = {Oct},\n  url     = {https://eugeneyan.com/writing/aligneval/}\n}\n
\n

Share on:

\n

\"\"

\n

\"\"

\n

\"\"

\n

\"\"

\n

\"\"

\n

Join 11,800+ readers getting updates on machine learning, RecSys, LLMs, and engineering.

\n", "content_markdown": "Go to [aligneval.com](https://aligneval.com) to start building your own LLM-evaluator; sample data included.\n\n> **Update**: Now [integrated with LangSmith](https://blog.langchain.com/introducing-align-evals/) and [featured on VentureBeat](https://venturebeat.com/ai/langchains-align-evals-closes-the-evaluator-trust-gap-with-prompt-level-calibration)! 🎉\n\nEvery AI-powered product needs evals. But let’s face it—they’re a pain to build, hard to scale, and most teams get them wrong. As a result, many AI-powered experiences are bottlenecked on evals, sometimes delaying launches by weeks or even months.\n\nI’ve spent the past year or so wrestling with product/task-specific evals. Testing different ways to [detect](https://eugeneyan.com/writing/abstractive/) hallucinations, [finetune](https://eugeneyan.com/writing/finetuning/) evaluators, and [evaluate](https://eugeneyan.com/writing/llm-evaluators/) LLM-based evaluators. There were dead ends. There were rabbit holes. But I learned [what works and what doesn’t](https://eugeneyan.com/writing/evals/).\n\nAnd that’s why I’m excited to introduce [AlignEval](https://aligneval.com/), an app that makes evals easy and… fun? (Okay, I’ll settle for less painful.) It also tries to automate part of the process. AlignEval makes building LLM-evaluators as straightforward as four simple steps:\n\n- Upload a CSV file with columns for input and output.\n- Look at the data and label samples as pass or fail.\n- Define eval criteria, run the LLM-evaluator, check results.\n- Optimize the LLM-evaluator with dev-test splits.\n\nBy the way, if you want to learn more about evals, my friends Hamel and Shreya are hosting their *final* cohort of “AI Evals for Engineers and PMs” in July. Here’s a [35% discount code](https://maven.com/parlance-labs/evals?promoCode=eugene-is-all-you-need).\n\n## Align AI to human. Calibrate human to AI. Repeat.\n\nThe key insight is that aligning AI to human preferences is only half the battle. To build effective evals, we must also **calibrate human criteria to AI output**.\n\nMany teams make the mistake of crafting elaborate eval criteria without first **looking at the data**. It’s like theorizing about user experience and defects from the ivory tower, without doing error analysis. From [Who Validates the Validators](https://arxiv.org/abs/2404.12272): “*It is impossible to completely determine evaluation criteria prior to human judging of LLM outputs.*”\n\nThis leads to two types of bad criteria. First, irrelevant criteria that are a waste of time, such as generic metrics (e.g., helpfulness) or very low probability defects (e.g., grammar, spelling). Second, unrealistic, unattainable criteria that the technology isn’t ready for, such as autonomous agents back in 2023. Either way, teams squander effort that would have been better invested in evaluating actual defects that occur with moderate frequency.\n\n(I think out-of-the-box eval solutions fuel this problem, pushing generic criteria that are easy to plug and play but ignore the idiosyncrasies of your domain, product, and data.)\n\nThe way to solve this—and build useful evals—is to work backward from the data. That’s what [AlignEval](https://aligneval.com/) tries to do by putting data and metrics first, not preset criteria or LLMs.\n\n## Getting started: Upload some data\n\n**To get started, upload a CSV file containing the following columns:**\n\n- **id**: Lets you match labeled and evaluated samples to your data\n- **input**: The context provided to the LLM to generate the output (e.g., text to classify, news to summarize, retrieved documents for answering questions)\n- **output**: What the LLM generates (e.g., classification label, summary, answer)\n- **label**: Your judgment on whether the output passes (0) or fails (1)\n\n![Instructions to upload a CSV file.](https://eugeneyan.com/assets/upload.webp \"Instructions to upload a CSV file.\")\n\nInstructions to upload a CSV file.\n\nIf you don’t have a CSV handy, [download a sample](https://aligneval.com/data/fib50.csv) based on the [Factual Inconsistency Benchmark](https://huggingface.co/datasets/r-three/fib). It contains 50 news articles, each with two summaries. One is factually consistent while the other is not. Perfect for AlignEval’s binary labeling task.\n\nAfter uploading the CSV, you’ll see the data in a flexible table view. Adjust the width of the browser to control how wide or narrow you want the table and columns to be.\n\n![How the table view looks after uploading a CSV file.](https://eugeneyan.com/assets/table.webp \"How the table view looks after uploading a CSV file.\")\n\nHow the table view looks after uploading a CSV file.\n\n## Labeling mode: Look at the data\n\n**In labeling mode, our only job is to look at the data.** AlignEval simplifies the process to a comparison between input and output fields. And for each sample, we only have to make a binary decision—pass or fail.\n\n![Instructions for labeling mode.](https://eugeneyan.com/assets/labeling.webp \"Instructions for labeling mode.\")\n\nInstructions for labeling mode.\n\nWhy binary labels? It leads to more accurate data, takes less time, and keeps cognitive load low. [DoorDash adopted a similar approach](https://eugeneyan.com/writing/content-moderation/#collecting-ground-truth-via-human-in-the-loop). Similarly, [Llama2](https://arxiv.org/abs/2307.09288) focused on collecting binary human preferences. The Llama2 authors also [shared at a meetup](https://www.youtube.com/live/CzR3OrOkM9w?si=gTrb6M44O941wrqb&t=913) that collecting binary preferences was much faster than writing samples for supervised finetuning.\n\nWhat’s important is that we look at the data with an open mind, and not write criteria based on our priors. We should also resist the urge to prematurely define criteria. We need to first immerse ourselves in the data. This reduces the risk of hunting for nonexistent defects or chasing unrealistic expectations. By understanding what the LLM actually generates, we can define more meaningful, better-calibrated criteria.\n\n[YouTube video player](https://www.youtube.com/watch?v=nnhMnNh_ZvU)\n\nLook at your data remix, courtesy of Jason Liu. Sign up for his next RAG course [here](https://maven.com/s/course/b9b1f32583).\n\nAfter labeling 20 rows, we unlock evaluation mode. (While evaluation mode is available after 20 labels, I think it’s too little to understand the data well enough to define criteria, and too small a sample for evaluation. I suggest aiming for 50-100 before writing criteria and running evals. The more familiar you are with your data, the better your evals will be.)\n\n## Evaluation mode: Write criteria, evaluate the evaluator\n\n**After unlocking evaluation mode, we can now write our task-specific evaluation criteria.** Keep it simple: Evaluate on a single dimension and return either 0 (pass) or 1 (fail).\n\n![Instructions for evaluation mode.](https://eugeneyan.com/assets/evaluation.webp \"Instructions for evaluation mode.\")\n\nInstructions for evaluation mode.\n\nHere’s the prompt I use for factual inconsistency classification. We start with only two sentences that define what a pass or fail looks like. We can refine it after running the LLM-evaluator and examining the explanations and predictions.\n\n![Writing the prompt and selecting models and fields.](https://eugeneyan.com/assets/prompt.webp \"Writing the prompt and selecting models and fields.\")\n\nWriting the prompt and selecting models and fields.\n\nNext, we choose our model (gpt-4o-mini or claude-3-haiku) and input fields (both input and output, or output only). For most tasks, (e.g., classification, summarization, Q&A), we want to compare the output to the input. Nonetheless, for some tasks, like evaluating style guide adherence or tone of voice, we only need the output. Focusing solely on the output reduces token cost and latency while potentially improving performance.\n\n**It’s also crucial to evaluate our LLM-evaluators against the labeled data.** Hit “Evaluate” and we’ll see metrics in the top right corner: sample size, recall, precision, F1, Cohen’s $\\\\kappa$, and counts for true and false positives/negatives.\n\n![Metrics to evaluate the LLM-evaluator.](https://eugeneyan.com/assets/metrics.webp \"Metrics to evaluate the LLM-evaluator.\")\n\nMetrics to evaluate the LLM-evaluator.\n\nIf our sample size is low (e.g., 20), the metrics can fluctuate a lot, even when we rerun the same LLM-evaluator. This is due to LLM stochasticity. We can improve stability by labeling more data. Aim for at least 50 labeled samples which also unlocks optimization mode.\n\n## Optimization mode: Semi-automated improvements 🤞\n\n**Optimization mode is where the magic (hopefully) happens.** Click “Optimize” and let AlignEval do its thing to improve your LLM-evaluator. While it’s still in early beta, it’s achieved decent improvements for several use cases. (And no, it’s not dspy.)\n\n![Instructions for optimization mode.](https://eugeneyan.com/assets/optimization.webp \"Instructions for optimization mode.\")\n\nInstructions for optimization mode.\n\nUnder the hood, optimization splits the labeled data into dev and test sets. (Since we’re not training a model, we call it a development split instead of a training split.) It then runs $n$ trials on the dev set to improve F1, using the same LLM and fields from evaluation mode. After $n$ trials, the improved LLM-evaluator is evaled on the unseen test set. In the table below, F1 starts at 0.571, increases to 0.722 at trial 5 on dev, then gets F1=0.727 on test.\n\n![Metrics from optimization trials on all, dev, and test splits.](https://eugeneyan.com/assets/optimize-table.webp \"Metrics from optimization trials on all, dev, and test splits.\")\n\nMetrics from optimization trials on all, dev, and test splits.\n\nSometimes, the dev and test metrics diverge significantly. With the small sample size (25 per split, 50 in total), this likely stems from the dev split being unrepresentative of the full data distribution, including the test split. Thus, the LLM-evaluator may overfit on the dev set and fail to generalize to the test set.\n\nTo improve generalization across the dev and test splits, we can label more data, ensuring that it’s diverse and representative. To be diverse is to be balanced in both 0 and 1 labels, and the variety of inputs and outputs. To be representative is to be similar to real-world examples. We can also run multiple LLM-evaluators in parallel and ensemble their scores. [PoLL](https://arxiv.org/abs/2404.18796) showed that an ensemble of three smaller LLM-evaluators outperformed gpt-4.\n\nFor a visual walkthrough, here’s a demo where I look at and label 50 rows of data lol.\n\n[YouTube video player](https://www.youtube.com/watch?v=OJJ7aHIBtPw)\n\n## Behind the scenes: How AlignEval was built\n\nAs a frontend development newbie (this is my first TypeScript app), I relied on the wisdom of the crowd, polling [Twitter](https://x.com/eugeneyan/status/1828447283811402006) and [LinkedIn](https://www.linkedin.com/posts/eugeneyan_if-you-were-building-a-small-web-app-today-activity-7234224078675439617-HX75/) for advice. I also [built the same app five times](https://eugeneyan.com/writing/web-frameworks/) to get a feel for several frameworks. Specifically, I tried FastHTML, Next.js, SvelteKit, and FastAPI + HTML (my go-to for prototyping).\n\nFrom early prototypes, Next.js seemed to be a good fit. Intuitive, scalable, and the perfect excuse to learn TypeScript. And thanks to Cursor, my beginner questions were answered easily: What’s the difference between `const` and `let`? What is a `prop`? It also made it easier to build UX components and fix bugs 2-3x faster, keeping velocity and motivation high.\n\nFor the backend, Python + FastAPI was the obvious choice. Python’s ecosystem of data analysis and machine learning made it easier to experiment with optimization mode.\n\nWhen it came to LLMs, I went with the smallest models from the biggest labs: gpt-4o-mini and claude-3-haiku (soon to be claude-3.5-haiku). They’re cheap, fast, and good enough for binary classification LLM-evaluators. It’s the perfect combo for a free app. And with each release, these small models just keep getting more and more capable.\n\nFor gpt-4o-mini, I used the [structured output](https://platform.openai.com/docs/guides/structured-outputs) functionality that’s still in beta. It allows users to define the desired output schema using Zod (JavaScript) or Pydantic (Python). Here’s an example in Javascript and its equivalent in Python:\n\n```javascript\nimport { z } from \"zod\";\n\nconst EvaluationResponse = z.object({\n explanation: z.string(),\n prediction: z.string().refine((val) => val === '0' || val === '1')\n});\n```\n\n```python\nfrom pydantic import BaseModel\n\nclass EvaluationResponse(BaseModel):\n explanation: str\n prediction: str\n```\n\nFor claude-3-haiku, I [defined the XML output](https://eugeneyan.com/writing/prompting/#structured-input-and-output) via the prompt. From what I’ve seen on the job and in AlignEval, this works 99.9% of the time. Here’s the prompt I used for AlignEval.\n\n```plaintext\nEvaluate the output based on the provided criteria. First, in the provided, think through your evaluation step by step.\n\nThen, provide a binary prediction (0 or 1) within .\n```\n\nTo decide on hosting, I conducted a [Twitter poll](https://x.com/eugeneyan/status/1828450401311764741) and consulted documentation for options like Dokku, Coolify, Vercel, etc. [Hamel’s write-up on Dokku](https://hamel.dev/blog/posts/dokku/) was particularly helpful. While I initially considered hosting the app on Hetzner with Dokku (another opportunity to learn something new), I ultimately decided to take the lazy path (read: prioritize ruthlessly) and start with hosting on Railway. It was a good decision.\n\nHosting on Railway has been a breeze. After setting up the project via their web UI and linking it via the CLI (`railway link`), pushing new updates is as simple as `railway up`. Each update takes one to two minutes. Railway also comes with several database options, including Postgres and Redis. While a VPS might be cheaper in the long run, at this stage, [YAGNI](https://en.wikipedia.org/wiki/You_aren%27t_gonna_need_it). Fiddling with a VPS would have been a distraction from improving the UX, implementing features, and figuring out optimization mode.\n\n![My simple Railway setup.](https://eugeneyan.com/assets/railway.webp \"My simple Railway setup.\")\n\nMy simple Railway setup\n\n> If you’d like to get started with Railway, please use my [affiliate code](https://railway.app?referralCode=fMY11k)—it’ll help with hosting costs and keep AlignEval running!\n\nGoing from a laptop prototype to a public beta took some effort. The work included:\n\n- Splitting into Next.js frontend, FastAPI backend, migrating from SQLite to Postgres\n- Implementing real-time feedback for labeling, evaluation, and optimization\n- Adding features to simplify labeling (pass/fail buttons), customize models and fields (dropdowns, integration with model providers), and more\n- Improving the UI (as rough as it looks now, it was way worse before lol)\n- Squashing lots of bugs (and probably missing many more)\n\nHere’s the to-do list I had; there were many more tasks and bugs that I neglected to track. Because I timeboxed the effort to October, a few items were not completed. But if AlignEval gains traction and proves useful, I might revisit the list.\n\n```markdown\n## Todos\n- [x] Split deployment into frontend and backend\n - https://help.railway.app/questions/how-to-expose-a-fast-api-backend-service-a1712631\n - https://help.railway.app/questions/econnrefused-when-calling-another-servic-5662d969#p-1\n - ~~[ ] Or consider something like this: https://vercel.com/templates/next.js/nextjs-fastapi-starter~~ (only works with Vercel)\n- [x] Add external database (Railway Postgres)\n- [x] Write interface to ensure compatibility between SQLite and Postgres\n- [x] Connect label-app to Railway Postgres\n- [x] Connect label-hpo to Railway Postgres\n- [x] Show feedback to users while optimization is running\n- [x] Update table names to use CSV file name\n- [x] Add support for multiple tables\n- [x] Replace tablename parameter with filename parameter\n- [x] Uploading a CSV file with an existing name should return the existing table\n- [x] Add llm-evaluator optimization to label-hpo\n- [x] Add dropdown to select evaluation fields\n- [x] Add ability to delete optimization table, with password\n- [x] Add progress bar to gamify labeling, evaluation and optimization\n- [x] Add buttons to click pass and fail\n- [x] Add usage count (upload (files, rows), evaluate (files, rows), optimize (files, trials))\n- [x] Add info button intro next to header to explain the site\n- [x] Update favicon\n- [x] Add support for OpenAI models\n- [x] Add balanced val-test split for optimization\n- [x] Add columns to optimization metrics: split (dev, test), model, evaluation_fields\n- [x] Set max upload rows to env variable\n- [ ] Fix bug where alert pops up: \"Failed to handle optimization: Unexpected token '<', \" Yan, Ziyou. (Oct 2024). AlignEval: Building an App to Make Evals Easy, Fun, and Automated. eugeneyan.com. https://eugeneyan.com/writing/aligneval/.\n\nor\n\n```\n@article{yan2024aligneval,\n title = {AlignEval: Building an App to Make Evals Easy, Fun, and Automated},\n author = {Yan, Ziyou},\n journal = {eugeneyan.com},\n year = {2024},\n month = {Oct},\n url = {https://eugeneyan.com/writing/aligneval/}\n}\n```\n\nShare on:\n\n![](https://eugeneyan.com/assets/icon-twitter.svg)\n\n![](https://eugeneyan.com/assets/icon-linkedin.svg)\n\n![](https://eugeneyan.com/assets/bluesky.svg)\n\n![](https://eugeneyan.com/assets/icon-facebook.svg)\n\n![](https://eugeneyan.com/assets/icon-mail.svg)\n\nJoin **11,800+** readers getting updates on machine learning, RecSys, LLMs, and engineering.\n", "dateCreated": "2026-09-27T19:29:15.293681927+00:00", "date_created": "2024-10-27T00:00:00+00:00", "date_modified": "2024-10-27T00:00:00+00:00", "date_published": "2024-10-27T00:00:00+00:00", "description": "Look at and label your data, build and evaluate your LLM-evaluator, and optimize it against your labels.", "external_url": "https://eugeneyan.com//writing/aligneval/", "git": { "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e", "path": "items/eugeneyan-com/2024/10/2024-10-27-aligneval-building-an-app-to-make-evals-easy-fun-and.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/eugeneyan-com/2024/10/2024-10-27-aligneval-building-an-app-to-make-evals-easy-fun-and.md" }, "id": "urn:aggr:item:bd0d843db43fc4688f8986b70239512785224d2e", "isBasedOn": { "@id": "https://eugeneyan.com//writing/aligneval/" }, "mainEntity": { "@id": "https://eugeneyan.com//writing/aligneval/", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/eugeneyan-com/2024-10-27-aligneval-building-an-app-to-make-evals-easy-fun-and/#webpage" }, "dateModified": "2024-10-27T00:00:00+00:00", "datePublished": "2024-10-27T00:00:00+00:00", "headline": "AlignEval: Building an App to Make Evals Easy, Fun, and Automated", "timeRequired": "PT13M", "url": "https://eugeneyan.com//writing/aligneval/", "wordCount": 2761 }, "name": "Archived snapshot: AlignEval: Building an App to Make Evals Easy, Fun, and Automated", "preview": null, "reading_minutes": 13, "replicated_at": null, "source": "eugeneyan-com", "source_name": "Eugene Yan", "summary": "Look at and label your data, build and evaluate your LLM-evaluator, and optimize it against your labels.", "tags": [ "eval", "learning", "llm", "🛠", "🩷" ], "temporalCoverage": "2026-09-27T19:29:15.293681927+00:00", "title": "AlignEval: Building an App to Make Evals Easy, Fun, and Automated", "url": "https://aggr.aymericbeaumet.com/items/eugeneyan-com/2024-10-27-aligneval-building-an-app-to-make-evals-easy-fun-and/", "word_count": 2761 }