{
  "@context": "https://schema.org",
  "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2025-10-07-bigcodearena-judging-code-generations-end-to-end-with-code/#webpage",
  "@type": [
    "WebPage",
    "ArchiveComponent"
  ],
  "articleBody": "Evaluating the quality of AI-generated code is notoriously difficult. While humans can easily spot whether a piece of code \"looks right,\" determining if it actually works correctly, handles edge cases properly, and produces the intended result requires running and testing it. This is why today, we're thrilled to announce **BigCodeArena** -- the first human-in-the-loop platform for evaluating code generation models through execution.\n\nInspired by LMArena for LLMs, we've built a platform that allows anyone to compare code generation models side-by-side, but with a crucial difference: **you can actually run the code and see what it produces**. Just submit a coding task, watch two different models generate solutions, execute both programs, and vote on which model produced better results. The outcomes are organized into a leaderboard that displays the community's highest-rated models.\n\n## Motivation\n\nThe field of code generation has long struggled with reliable evaluation methods. Traditional benchmarks like HumanEval test code against predefined test cases, but these represent only a tiny fraction of real-world programming tasks. Human evaluation platforms exist for general chatbots, but they fall short for code: reading raw source code and mentally simulating its execution is cognitively demanding and error-prone, especially for longer programs or complex UI applications.\n\nConsider this scenario:\n\n> You ask two AI models to build a responsive photo gallery website. Both generate code that looks syntactically correct. But which one is actually better? Without running the code, it's nearly impossible to tell. One might produce a beautiful, functional grid layout, while the other might have subtle bugs or poor styling that only become apparent when rendered in a browser.\n\n[![bigcodearena-demo](https://cdn-uploads.huggingface.co/production/uploads/62b7fb545233925f253531c8/SspOHYTm-BT-zIlIV3Rdg.png)](https://cdn-uploads.huggingface.co/production/uploads/62b7fb545233925f253531c8/SspOHYTm-BT-zIlIV3Rdg.png)\n\nThis observation led us to a key insight: **execution feedback is essential for humans to judge code quality reliably.** That's exactly what BigCodeArena provides.\n\n## The BigCodeArena Platform\n\nBigCodeArena extends the Chatbot Arena framework with powerful features specifically designed for code evaluation:\n\n### Real-Time Execution\n\nEvery code snippet generated by models is automatically executed in isolated sandbox environments. Whether it's a Python script, a React web app, a PyGame game, or a C++ algorithm, you can see the actual output, not just the source code.\n\n### Multi-Language & Framework Support\n\nWe currently support 10 languages (Python, JavaScript, TypeScript, HTML, C, C++, Java, Go, Rust, and Markdown) and 8 execution environments:\n\n- **Web Frameworks**: React, Vue, Core Web (vanilla HTML/CSS/JS)\n- **Python Frameworks**: Streamlit, Gradio, PyGame\n- **Diagrams**: Mermaid\n- **General Purpose Interpreters**: Python and JavaScript code interpreters, plus compiled language runners\n\n### Interactive Testing\n\nUnlike static code comparison, you can actually interact with the generated applications:\n\n- Click buttons and test UI elements in web apps\n- Play the games generated by models\n- Edit the code and re-run it to test modifications\n- View visual outputs like plots, charts, and diagrams\n\n[![bigcodearena](https://cdn-uploads.huggingface.co/production/uploads/62b7fb545233925f253531c8/2AvLXvIP1VE1moRw9BHS_.png)](https://cdn-uploads.huggingface.co/production/uploads/62b7fb545233925f253531c8/2AvLXvIP1VE1moRw9BHS_.png)\n\n### Multi-Turn Conversations\n\nReal programming isn't one-and-done. BigCodeArena supports multi-turn interactions, allowing you to refine requirements, ask for features to be added, or request bug fixes -- just like working with a real coding assistant.\n\n## What We've Learned: 5 Months of Community Evaluation\n\nSince launching in February 2025, BigCodeArena has collected over **14,000 conversations** from more than 500 unique users, with **4,700+ high-quality preference votes** comparing 10 frontier LLMs.\n\n### Programming Topics in the Wild\n\nOur users have explored remarkably diverse coding scenarios:\n\n- **Web Design (36%)**: Building responsive websites, interactive dashboards, and web applications\n- **Problem Solving (23%)**: Algorithms, data structures, and computational challenges\n- **Game Development (16%)**: Creating interactive games with physics, collision detection, and graphics\n- **Scientific Computing (14%)**: Data analysis, visualization, and numerical simulations\n- **Creative Coding (8%)**: Artistic visualizations, generative art, and experimental interfaces\n- **Diagram Creation (3%)**: Flowcharts, system architectures, and data visualizations\n\n### Language and Framework Popularity\n\nPython dominates with over 4,000 conversations, followed by JavaScript/TypeScript (3,359), HTML (1,601), and C++ (642). Among frameworks, direct Python interpreters lead usage (6,000 sessions), with React (2,729), Core Web (1,574), Streamlit (1,254), and PyGame (1,087) also seeing heavy use.\n\n### User Interaction Patterns\n\nMost interactions are focused and efficient: 76% of conversations consist of just 2 turns (one request, one response), with a mean conversation length of 4.12 messages. However, the platform supports extended multi-turn debugging sessions when needed, with some conversations exceeding 10 turns as users refine complex applications.\n\n### Model Rankings from Community Votes\n\nFrom our 14K conversations, we filtered for high-quality pairwise comparisons: conversations with at least two turns and actual code execution. This yielded **4,731 voting samples**, with each evaluated model receiving at least 700 votes. We aggregate these votes into **Elo ratings** using the Bradley-Terry model, which estimates the probability that one model beats another based on head-to-head comparisons.\n\nTo ensure robust rankings, we use 100 bootstrap resamples to construct 95% confidence intervals, so we can identify statistically significant performance differences between models.\n\n[![elo_ratings_comparison_page-0001](https://cdn-uploads.huggingface.co/production/uploads/62b7fb545233925f253531c8/rEnROVN53CxyMHsLh223J.jpeg)](https://cdn-uploads.huggingface.co/production/uploads/62b7fb545233925f253531c8/rEnROVN53CxyMHsLh223J.jpeg)\n\nWe evaluate models under three settings to control for different factors:\n\n1. **All Data**: Uses all pairwise comparisons regardless of execution environment or programming language\n2. **Environment Matched**: Only compares models when both were executed in the same sandbox (e.g., both in React or both in PyGame)\n3. **Language Matched**: Further restricts comparisons to the same programming language\n\nRankings remain remarkably consistent across all three settings, revealing clear performance tiers:\n\n**Top Tier**: **o3-mini** and **o1-mini** consistently lead with the highest Elo ratings and tight confidence intervals. These models maintain top performance regardless of environment or language constraints, showing strong robustness across coding scenarios. **Claude-3.5-Sonnet** follows closely, particularly excelling when language is controlled.\n\n**Mid Tier**: **GPT-4o**, **o1**, and **Gemini-2.0-Pro/Flash** form a competitive middle tier. GPT-4o shows some sensitivity to language matching, suggesting room for improvement in multilingual consistency.\n\n**Open Source Models**: **Qwen2.5** variants and **Llama-3.3-70B** lag behind frontier proprietary models, highlighting the performance gap that remains between leading closed and open models.\n\n[![combined_performance_heatmaps_page-0001](https://cdn-uploads.huggingface.co/production/uploads/62b7fb545233925f253531c8/5iB4XRwOBzjEdQM5tqVoU.jpeg)](https://cdn-uploads.huggingface.co/production/uploads/62b7fb545233925f253531c8/5iB4XRwOBzjEdQM5tqVoU.jpeg)\n\n*Figure: Overall win rate heatmaps (percentage of all pairwise comparisons won) of each model in the sessions across languages (left) and execution environments (right). For each category, we only keep models that appear in at least 3 conversation sessions.*\n\n#### Performance Across Languages\n\nBreaking down performance by programming language reveals interesting patterns:\n\n- Top-tier models like **o3-mini** and **o1-mini** achieve dominant win rates in mainstream languages like Python, Java, and C++\n- **Gemini-2.0-Pro** shows particular strength in Rust, achieving the highest win rate in that category\n- Different models exhibit distinct areas of expertise, with frontier models excelling in different niches\n- Open models like Qwen2.5 variants show inconsistent performance, particularly struggling with Rust and Go\n\n#### Performance Across Execution Environments\n\nAnalyzing win rates by execution environment reveals how models handle different runtime contexts:\n\n**Robust Performers**: **o3-mini** maintains consistently strong performance across React, Streamlit, Gradio, Core Web, and PyGame, demonstrating excellent environmental adaptability.\n\n**Stable but Selective**: **Claude-3.5-Sonnet** and **Gemini-2.0-Flash** show generally stable performance but with reduced win rates in complex UI-heavy environments like Vue and Mermaid.\n\n**Framework-Specific Weaknesses**: **Qwen2.5** models, while competitive in some web frameworks (Core Web, React), struggle significantly with interactive and visualization-oriented environments like PyGame, Vue, and Mermaid. These environments often require precise handling of control flow, graphics rendering, and package dependencies.\n\nThese results highlight an important insight: **aggregate Elo scores don't tell the whole story**. Some models remain brittle under specific runtime constraints, and execution environment matters significantly for real-world deployment.\n\n## Two New Benchmarks: BigCodeReward and AutoCodeArena\n\nTo advance research beyond crowdsourced evaluation, we're releasing two complementary benchmarks:\n\n### BigCodeReward: Evaluating Reward Models for Code\n\nBuilding on our 4,700+ preference votes, BigCodeReward tests how well LLMs can judge code quality when acting as reward models. The key finding? **Execution results dramatically improve judgment accuracy.**\n\nWhen models can see execution outputs (screenshots of web apps, game visuals, program logs), their alignment with human preferences increases substantially:\n\n- Claude-Sonnet-4: **56.7% → 62.3%** accuracy\n- GPT-4o: **54.6% → 63.8%** accuracy\n- Qwen2.5-VL-72B: **58.7% → 66.2%** accuracy\n\nThis reinforces our core thesis: you can't reliably judge code without running it -- and this applies to both humans *and* AI judges.\n\n### AutoCodeArena: Automated Code Generation Benchmarks\n\nInspired by Arena-Hard-Auto, AutoCodeArena provides a scalable way to evaluate new models without waiting for thousands of human votes. We carefully selected 600 representative prompts from our crowdsourced data, spanning all programming topics and frameworks.\n\nUsing automated LLM judges (Claude-3.7-Sonnet) to evaluate code execution results against a GPT-4.1 baseline, we can rapidly benchmark new models. This approach enables weekly leaderboard updates as new models are released.\n\nOur automated benchmark evaluated 20+ cutting-edge models, including recently released systems:\n\n**Top Performers:**\n\n1. **GPT-5** -- Establishes new state-of-the-art by a significant margin\n2. **Claude-Opus-4** and **Claude-Sonnet-4** -- Strong second tier, excelling in reasoning-heavy tasks\n3. **Qwen3-Coder**, **Kimi-K2**, **GLM-4.5** -- Leading open models that narrow the gap with mid-tier proprietary systems\n\n[![overall_performance](https://cdn-uploads.huggingface.co/production/uploads/62b7fb545233925f253531c8/NzyobCl-pYwgR-6ggufMe.jpeg)](https://cdn-uploads.huggingface.co/production/uploads/62b7fb545233925f253531c8/NzyobCl-pYwgR-6ggufMe.jpeg)\n\n*Figure: Win rates of recent LLMs on AutoCodeArena against a GPT-4.1 baseline, judged by Claude-3.7-Sonnet. The 50% mark represents parity with GPT-4.1. Models above this line outperform the baseline, while those below underperform. Error bars show 95% confidence intervals. Note: Claude-3.7-Sonnet is excluded from rankings to avoid self-judgment bias, and GPT-4.1 appears only as the reference baseline.*\n\nThe results show that while proprietary models maintain an edge, open-source models are rapidly closing the gap, with some approaching GPT-4.1-level performance.\n\n## Try It Yourself\n\nBigCodeArena is open to everyone -- no account required! Visit [https://huggingface.co/spaces/bigcode/arena](https://huggingface.co/spaces/bigcode/arena) to:\n\n- Compare code from more recent frontier LLMs (e.g., Qwen3, DeepSeek-V3.X, and other proprietary models)\n- Test web apps, games, visualizations, and algorithms\n- See real execution results, not just source code\n- Vote on your preferences to help improve the leaderboard\n- Explore multi-turn coding conversations\n\nWhether you're building a React dashboard, creating a PyGame game, solving algorithmic challenges, or generating creative visualizations, BigCodeArena lets you see which models truly deliver.\n\n## Open Source Everything\n\nFollowing the BigCode Project's commitment to transparency, we're releasing:\n\n- **Codebase**: Full evaluation pipelines and Gradio application source ([GitHub](https://github.com/bigcode-project/bigcodearena))\n- **Crowdsourced Data**: 14K raw conversations and 4.7K preference votes ([HuggingFace Collection](https://huggingface.co/collections/bigcode/bigcodearena-68cd3a196e5147cc45f8ea3d))\n- **Benchmarks**: BigCodeReward and AutoCodeArena datasets\n\n## What's Next?\n\nWe envision BigCodeArena as a long-term project that evolves with the community:\n\n- **Expanded Language Support**: More programming languages and frameworks.\n- **Live Benchmarks**: Continuously refreshed evaluation prompts to prevent overfitting\n- **Agent-Based Evaluation**: Using AI agents to interact with web apps for deeper testing\n- **Better Reward Models**: Advancing automated code quality assessment\n- **Community Contributions**: We welcome new execution environments, evaluation criteria, and model additions. [PRs](https://huggingface.co/spaces/bigcode/arena/discussions?new_pr=true) are always welcome!\n\n## Conclusion\n\nEvaluating code isn't like evaluating text -- you need to run it, test it, and interact with it. BigCodeArena makes this possible at scale, combining human judgment with real execution feedback to create the most reliable evaluation platform for code generation models.\n\nJoin us in building the future of code generation evaluation. Write a prompt, compare the models, and vote for your favorite. Your feedback helps the entire community understand which models truly deliver on the promise of AI-assisted programming.\n\nWe'd love to hear your feedback! Connect with us on [GitHub](https://github.com/bigcode-project/bigcodearena), join discussions in the [Hugging Face Space community tab](https://huggingface.co/spaces/bigcode/arena/discussions), or reach out to the BigCode Project at `contact@bigcode-project.org`.\n\n## Acknowledgements\n\nWe thank [Leandro von Werra](https://huggingface.co/lvwerra) for his valuable suggestions and feedback on the blog.\n\n## Citation\n\n```bibtex\n@article{zhuo2025bigcodearena,\n    title={BigCodeArena: Unveiling More Reliable Human Preferences in Code Generation via Execution},\n    author={Terry Yue Zhuo, Xiaolong Jin, Hange Liu, Juyong Jiang, Tianyang Liu, Chen Gong, Bhupesh Bishnoi, Vaisakhi Mishra, Marek Suppa, Noah Ziems, Saiteja Utpala, Ming Xu, Guangyu Song, Kaixin Li, Yuhan Cao, Bo Liu, Zheng Liu, Sabina Abdurakhmanova, Wenhao Yu, Mengzhao Jia, Jihan Yao, Kenneth Hamilton, Kumar Shridhar, Minh Chien Vu, Dingmin Wang, Jiawei Liu, Zijian Wang, Qian Liu, Binyuan Hui, Meg Risdal, Ahsen Khaliq, Atin Sood, Zhenchang Xing, Wasi Uddin Ahmad, John Grundy, David Lo, Banghua Zhu, Xiaoning Du, Torsten Scholak, Leandro von Werra},\n    year={2025}\n}\n```\n\n* * *\n\n**Try BigCodeArena now**: [Hugging Face Space](https://huggingface.co/spaces/bigcode/arena)\n\n**Read the paper**: [Hugging Face](https://huggingface.co/papers/2510.08697)\n\n**Run the code**: [GitHub](https://github.com/bigcode-project/bigcodearena)\n\n**Explore the collection**: [Hugging Face Collection](https://huggingface.co/collections/bigcode/bigcodearena-68cd3a196e5147cc45f8ea3d)\n",
  "authors": [],
  "captured_at": "2026-10-04T12:50:30.804479567+00:00",
  "category": "ai",
  "content_html": "<p>Evaluating the quality of AI-generated code is notoriously difficult. While humans can easily spot whether a piece of code &quot;looks right,&quot; determining if it actually works correctly, handles edge cases properly, and produces the intended result requires running and testing it. This is why today, we're thrilled to announce <strong>BigCodeArena</strong> -- the first human-in-the-loop platform for evaluating code generation models through execution.</p>\n<p>Inspired by LMArena for LLMs, we've built a platform that allows anyone to compare code generation models side-by-side, but with a crucial difference: <strong>you can actually run the code and see what it produces</strong>. Just submit a coding task, watch two different models generate solutions, execute both programs, and vote on which model produced better results. The outcomes are organized into a leaderboard that displays the community's highest-rated models.</p>\n<h2>Motivation</h2>\n<p>The field of code generation has long struggled with reliable evaluation methods. Traditional benchmarks like HumanEval test code against predefined test cases, but these represent only a tiny fraction of real-world programming tasks. Human evaluation platforms exist for general chatbots, but they fall short for code: reading raw source code and mentally simulating its execution is cognitively demanding and error-prone, especially for longer programs or complex UI applications.</p>\n<p>Consider this scenario:</p>\n<blockquote>\n<p>You ask two AI models to build a responsive photo gallery website. Both generate code that looks syntactically correct. But which one is actually better? Without running the code, it's nearly impossible to tell. One might produce a beautiful, functional grid layout, while the other might have subtle bugs or poor styling that only become apparent when rendered in a browser.</p>\n</blockquote>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://cdn-uploads.huggingface.co/production/uploads/62b7fb545233925f253531c8/SspOHYTm-BT-zIlIV3Rdg.png\"><picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/62b7fb545233925f253531c8/SspOHYTm-BT-zIlIV3Rdg.png\" alt=\"bigcodearena-demo\" loading=\"eager\" decoding=\"async\" fetchpriority=\"high\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture></a></p>\n<p>This observation led us to a key insight: <strong>execution feedback is essential for humans to judge code quality reliably.</strong> That's exactly what BigCodeArena provides.</p>\n<h2>The BigCodeArena Platform</h2>\n<p>BigCodeArena extends the Chatbot Arena framework with powerful features specifically designed for code evaluation:</p>\n<h3>Real-Time Execution</h3>\n<p>Every code snippet generated by models is automatically executed in isolated sandbox environments. Whether it's a Python script, a React web app, a PyGame game, or a C++ algorithm, you can see the actual output, not just the source code.</p>\n<h3>Multi-Language &amp; Framework Support</h3>\n<p>We currently support 10 languages (Python, JavaScript, TypeScript, HTML, C, C++, Java, Go, Rust, and Markdown) and 8 execution environments:</p>\n<ul>\n<li><strong>Web Frameworks</strong>: React, Vue, Core Web (vanilla HTML/CSS/JS)</li>\n<li><strong>Python Frameworks</strong>: Streamlit, Gradio, PyGame</li>\n<li><strong>Diagrams</strong>: Mermaid</li>\n<li><strong>General Purpose Interpreters</strong>: Python and JavaScript code interpreters, plus compiled language runners</li>\n</ul>\n<h3>Interactive Testing</h3>\n<p>Unlike static code comparison, you can actually interact with the generated applications:</p>\n<ul>\n<li>Click buttons and test UI elements in web apps</li>\n<li>Play the games generated by models</li>\n<li>Edit the code and re-run it to test modifications</li>\n<li>View visual outputs like plots, charts, and diagrams</li>\n</ul>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://cdn-uploads.huggingface.co/production/uploads/62b7fb545233925f253531c8/2AvLXvIP1VE1moRw9BHS_.png\"><picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/62b7fb545233925f253531c8/2AvLXvIP1VE1moRw9BHS_.png\" alt=\"bigcodearena\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture></a></p>\n<h3>Multi-Turn Conversations</h3>\n<p>Real programming isn't one-and-done. BigCodeArena supports multi-turn interactions, allowing you to refine requirements, ask for features to be added, or request bug fixes -- just like working with a real coding assistant.</p>\n<h2>What We've Learned: 5 Months of Community Evaluation</h2>\n<p>Since launching in February 2025, BigCodeArena has collected over <strong>14,000 conversations</strong> from more than 500 unique users, with <strong>4,700+ high-quality preference votes</strong> comparing 10 frontier LLMs.</p>\n<h3>Programming Topics in the Wild</h3>\n<p>Our users have explored remarkably diverse coding scenarios:</p>\n<ul>\n<li><strong>Web Design (36%)</strong>: Building responsive websites, interactive dashboards, and web applications</li>\n<li><strong>Problem Solving (23%)</strong>: Algorithms, data structures, and computational challenges</li>\n<li><strong>Game Development (16%)</strong>: Creating interactive games with physics, collision detection, and graphics</li>\n<li><strong>Scientific Computing (14%)</strong>: Data analysis, visualization, and numerical simulations</li>\n<li><strong>Creative Coding (8%)</strong>: Artistic visualizations, generative art, and experimental interfaces</li>\n<li><strong>Diagram Creation (3%)</strong>: Flowcharts, system architectures, and data visualizations</li>\n</ul>\n<h3>Language and Framework Popularity</h3>\n<p>Python dominates with over 4,000 conversations, followed by JavaScript/TypeScript (3,359), HTML (1,601), and C++ (642). Among frameworks, direct Python interpreters lead usage (6,000 sessions), with React (2,729), Core Web (1,574), Streamlit (1,254), and PyGame (1,087) also seeing heavy use.</p>\n<h3>User Interaction Patterns</h3>\n<p>Most interactions are focused and efficient: 76% of conversations consist of just 2 turns (one request, one response), with a mean conversation length of 4.12 messages. However, the platform supports extended multi-turn debugging sessions when needed, with some conversations exceeding 10 turns as users refine complex applications.</p>\n<h3>Model Rankings from Community Votes</h3>\n<p>From our 14K conversations, we filtered for high-quality pairwise comparisons: conversations with at least two turns and actual code execution. This yielded <strong>4,731 voting samples</strong>, with each evaluated model receiving at least 700 votes. We aggregate these votes into <strong>Elo ratings</strong> using the Bradley-Terry model, which estimates the probability that one model beats another based on head-to-head comparisons.</p>\n<p>To ensure robust rankings, we use 100 bootstrap resamples to construct 95% confidence intervals, so we can identify statistically significant performance differences between models.</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://cdn-uploads.huggingface.co/production/uploads/62b7fb545233925f253531c8/rEnROVN53CxyMHsLh223J.jpeg\"><picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/62b7fb545233925f253531c8/rEnROVN53CxyMHsLh223J.jpeg\" alt=\"elo_ratings_comparison_page-0001\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture></a></p>\n<p>We evaluate models under three settings to control for different factors:</p>\n<ol>\n<li><strong>All Data</strong>: Uses all pairwise comparisons regardless of execution environment or programming language</li>\n<li><strong>Environment Matched</strong>: Only compares models when both were executed in the same sandbox (e.g., both in React or both in PyGame)</li>\n<li><strong>Language Matched</strong>: Further restricts comparisons to the same programming language</li>\n</ol>\n<p>Rankings remain remarkably consistent across all three settings, revealing clear performance tiers:</p>\n<p><strong>Top Tier</strong>: <strong>o3-mini</strong> and <strong>o1-mini</strong> consistently lead with the highest Elo ratings and tight confidence intervals. These models maintain top performance regardless of environment or language constraints, showing strong robustness across coding scenarios. <strong>Claude-3.5-Sonnet</strong> follows closely, particularly excelling when language is controlled.</p>\n<p><strong>Mid Tier</strong>: <strong>GPT-4o</strong>, <strong>o1</strong>, and <strong>Gemini-2.0-Pro/Flash</strong> form a competitive middle tier. GPT-4o shows some sensitivity to language matching, suggesting room for improvement in multilingual consistency.</p>\n<p><strong>Open Source Models</strong>: <strong>Qwen2.5</strong> variants and <strong>Llama-3.3-70B</strong> lag behind frontier proprietary models, highlighting the performance gap that remains between leading closed and open models.</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://cdn-uploads.huggingface.co/production/uploads/62b7fb545233925f253531c8/5iB4XRwOBzjEdQM5tqVoU.jpeg\"><picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/62b7fb545233925f253531c8/5iB4XRwOBzjEdQM5tqVoU.jpeg\" alt=\"combined_performance_heatmaps_page-0001\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture></a></p>\n<p><em>Figure: Overall win rate heatmaps (percentage of all pairwise comparisons won) of each model in the sessions across languages (left) and execution environments (right). For each category, we only keep models that appear in at least 3 conversation sessions.</em></p>\n<h4>Performance Across Languages</h4>\n<p>Breaking down performance by programming language reveals interesting patterns:</p>\n<ul>\n<li>Top-tier models like <strong>o3-mini</strong> and <strong>o1-mini</strong> achieve dominant win rates in mainstream languages like Python, Java, and C++</li>\n<li><strong>Gemini-2.0-Pro</strong> shows particular strength in Rust, achieving the highest win rate in that category</li>\n<li>Different models exhibit distinct areas of expertise, with frontier models excelling in different niches</li>\n<li>Open models like Qwen2.5 variants show inconsistent performance, particularly struggling with Rust and Go</li>\n</ul>\n<h4>Performance Across Execution Environments</h4>\n<p>Analyzing win rates by execution environment reveals how models handle different runtime contexts:</p>\n<p><strong>Robust Performers</strong>: <strong>o3-mini</strong> maintains consistently strong performance across React, Streamlit, Gradio, Core Web, and PyGame, demonstrating excellent environmental adaptability.</p>\n<p><strong>Stable but Selective</strong>: <strong>Claude-3.5-Sonnet</strong> and <strong>Gemini-2.0-Flash</strong> show generally stable performance but with reduced win rates in complex UI-heavy environments like Vue and Mermaid.</p>\n<p><strong>Framework-Specific Weaknesses</strong>: <strong>Qwen2.5</strong> models, while competitive in some web frameworks (Core Web, React), struggle significantly with interactive and visualization-oriented environments like PyGame, Vue, and Mermaid. These environments often require precise handling of control flow, graphics rendering, and package dependencies.</p>\n<p>These results highlight an important insight: <strong>aggregate Elo scores don't tell the whole story</strong>. Some models remain brittle under specific runtime constraints, and execution environment matters significantly for real-world deployment.</p>\n<h2>Two New Benchmarks: BigCodeReward and AutoCodeArena</h2>\n<p>To advance research beyond crowdsourced evaluation, we're releasing two complementary benchmarks:</p>\n<h3>BigCodeReward: Evaluating Reward Models for Code</h3>\n<p>Building on our 4,700+ preference votes, BigCodeReward tests how well LLMs can judge code quality when acting as reward models. The key finding? <strong>Execution results dramatically improve judgment accuracy.</strong></p>\n<p>When models can see execution outputs (screenshots of web apps, game visuals, program logs), their alignment with human preferences increases substantially:</p>\n<ul>\n<li>Claude-Sonnet-4: <strong>56.7% → 62.3%</strong> accuracy</li>\n<li>GPT-4o: <strong>54.6% → 63.8%</strong> accuracy</li>\n<li>Qwen2.5-VL-72B: <strong>58.7% → 66.2%</strong> accuracy</li>\n</ul>\n<p>This reinforces our core thesis: you can't reliably judge code without running it -- and this applies to both humans <em>and</em> AI judges.</p>\n<h3>AutoCodeArena: Automated Code Generation Benchmarks</h3>\n<p>Inspired by Arena-Hard-Auto, AutoCodeArena provides a scalable way to evaluate new models without waiting for thousands of human votes. We carefully selected 600 representative prompts from our crowdsourced data, spanning all programming topics and frameworks.</p>\n<p>Using automated LLM judges (Claude-3.7-Sonnet) to evaluate code execution results against a GPT-4.1 baseline, we can rapidly benchmark new models. This approach enables weekly leaderboard updates as new models are released.</p>\n<p>Our automated benchmark evaluated 20+ cutting-edge models, including recently released systems:</p>\n<p><strong>Top Performers:</strong></p>\n<ol>\n<li><strong>GPT-5</strong> -- Establishes new state-of-the-art by a significant margin</li>\n<li><strong>Claude-Opus-4</strong> and <strong>Claude-Sonnet-4</strong> -- Strong second tier, excelling in reasoning-heavy tasks</li>\n<li><strong>Qwen3-Coder</strong>, <strong>Kimi-K2</strong>, <strong>GLM-4.5</strong> -- Leading open models that narrow the gap with mid-tier proprietary systems</li>\n</ol>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://cdn-uploads.huggingface.co/production/uploads/62b7fb545233925f253531c8/NzyobCl-pYwgR-6ggufMe.jpeg\"><picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/62b7fb545233925f253531c8/NzyobCl-pYwgR-6ggufMe.jpeg\" alt=\"overall_performance\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture></a></p>\n<p><em>Figure: Win rates of recent LLMs on AutoCodeArena against a GPT-4.1 baseline, judged by Claude-3.7-Sonnet. The 50% mark represents parity with GPT-4.1. Models above this line outperform the baseline, while those below underperform. Error bars show 95% confidence intervals. Note: Claude-3.7-Sonnet is excluded from rankings to avoid self-judgment bias, and GPT-4.1 appears only as the reference baseline.</em></p>\n<p>The results show that while proprietary models maintain an edge, open-source models are rapidly closing the gap, with some approaching GPT-4.1-level performance.</p>\n<h2>Try It Yourself</h2>\n<p>BigCodeArena is open to everyone -- no account required! Visit <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://huggingface.co/spaces/bigcode/arena\">https://huggingface.co/spaces/bigcode/arena</a> to:</p>\n<ul>\n<li>Compare code from more recent frontier LLMs (e.g., Qwen3, DeepSeek-V3.X, and other proprietary models)</li>\n<li>Test web apps, games, visualizations, and algorithms</li>\n<li>See real execution results, not just source code</li>\n<li>Vote on your preferences to help improve the leaderboard</li>\n<li>Explore multi-turn coding conversations</li>\n</ul>\n<p>Whether you're building a React dashboard, creating a PyGame game, solving algorithmic challenges, or generating creative visualizations, BigCodeArena lets you see which models truly deliver.</p>\n<h2>Open Source Everything</h2>\n<p>Following the BigCode Project's commitment to transparency, we're releasing:</p>\n<ul>\n<li><strong>Codebase</strong>: Full evaluation pipelines and Gradio application source (<a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://github.com/bigcode-project/bigcodearena\">GitHub</a>)</li>\n<li><strong>Crowdsourced Data</strong>: 14K raw conversations and 4.7K preference votes (<a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://huggingface.co/collections/bigcode/bigcodearena-68cd3a196e5147cc45f8ea3d\">HuggingFace Collection</a>)</li>\n<li><strong>Benchmarks</strong>: BigCodeReward and AutoCodeArena datasets</li>\n</ul>\n<h2>What's Next?</h2>\n<p>We envision BigCodeArena as a long-term project that evolves with the community:</p>\n<ul>\n<li><strong>Expanded Language Support</strong>: More programming languages and frameworks.</li>\n<li><strong>Live Benchmarks</strong>: Continuously refreshed evaluation prompts to prevent overfitting</li>\n<li><strong>Agent-Based Evaluation</strong>: Using AI agents to interact with web apps for deeper testing</li>\n<li><strong>Better Reward Models</strong>: Advancing automated code quality assessment</li>\n<li><strong>Community Contributions</strong>: We welcome new execution environments, evaluation criteria, and model additions. <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://huggingface.co/spaces/bigcode/arena/discussions?new_pr=true\">PRs</a> are always welcome!</li>\n</ul>\n<h2>Conclusion</h2>\n<p>Evaluating code isn't like evaluating text -- you need to run it, test it, and interact with it. BigCodeArena makes this possible at scale, combining human judgment with real execution feedback to create the most reliable evaluation platform for code generation models.</p>\n<p>Join us in building the future of code generation evaluation. Write a prompt, compare the models, and vote for your favorite. Your feedback helps the entire community understand which models truly deliver on the promise of AI-assisted programming.</p>\n<p>We'd love to hear your feedback! Connect with us on <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://github.com/bigcode-project/bigcodearena\">GitHub</a>, join discussions in the <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://huggingface.co/spaces/bigcode/arena/discussions\">Hugging Face Space community tab</a>, or reach out to the BigCode Project at <code>contact@bigcode-project.org</code>.</p>\n<h2>Acknowledgements</h2>\n<p>We thank <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://huggingface.co/lvwerra\">Leandro von Werra</a> for his valuable suggestions and feedback on the blog.</p>\n<h2>Citation</h2>\n<pre><code class=\"language-bibtex\"><span class=\"code-snippet\" data-language=\"Bibtex\"><span class=\"syntax-text syntax-bibtex\"><span class=\"syntax-meta syntax-entry syntax-braces syntax-bibtex\"><span class=\"syntax-keyword syntax-other syntax-entry-type syntax-bibtex\"><span class=\"syntax-punctuation syntax-definition syntax-keyword syntax-bibtex\">@</span>article</span><span class=\"syntax-punctuation syntax-section syntax-entry syntax-begin syntax-bibtex\">{</span><span class=\"syntax-entity syntax-name syntax-type syntax-entry-key syntax-bibtex\">zhuo2025bigcodearena</span>,\n    <span class=\"syntax-meta syntax-key-assignment syntax-bibtex\"><span class=\"syntax-string syntax-unquoted syntax-key syntax-bibtex\">title</span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-bibtex\">=</span><span class=\"syntax-string syntax-quoted syntax-other syntax-braces syntax-bibtex\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-bibtex\">{</span>BigCodeArena: Unveiling More Reliable Human Preferences in Code Generation via Execution<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-bibtex\">}</span></span></span>,\n    <span class=\"syntax-meta syntax-key-assignment syntax-bibtex\"><span class=\"syntax-string syntax-unquoted syntax-key syntax-bibtex\">author</span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-bibtex\">=</span><span class=\"syntax-string syntax-quoted syntax-other syntax-braces syntax-bibtex\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-bibtex\">{</span>Terry Yue Zhuo, Xiaolong Jin, Hange Liu, Juyong Jiang, Tianyang Liu, Chen Gong, Bhupesh Bishnoi, Vaisakhi Mishra, Marek Suppa, Noah Ziems, Saiteja Utpala, Ming Xu, Guangyu Song, Kaixin Li, Yuhan Cao, Bo Liu, Zheng Liu, Sabina Abdurakhmanova, Wenhao Yu, Mengzhao Jia, Jihan Yao, Kenneth Hamilton, Kumar Shridhar, Minh Chien Vu, Dingmin Wang, Jiawei Liu, Zijian Wang, Qian Liu, Binyuan Hui, Meg Risdal, Ahsen Khaliq, Atin Sood, Zhenchang Xing, Wasi Uddin Ahmad, John Grundy, David Lo, Banghua Zhu, Xiaoning Du, Torsten Scholak, Leandro von Werra<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-bibtex\">}</span></span></span>,\n    <span class=\"syntax-meta syntax-key-assignment syntax-bibtex\"><span class=\"syntax-string syntax-unquoted syntax-key syntax-bibtex\">year</span><span class=\"syntax-punctuation syntax-separator syntax-key-value syntax-bibtex\">=</span><span class=\"syntax-string syntax-quoted syntax-other syntax-braces syntax-bibtex\"><span class=\"syntax-punctuation syntax-definition syntax-string syntax-begin syntax-bibtex\">{</span>2025<span class=\"syntax-punctuation syntax-definition syntax-string syntax-end syntax-bibtex\">}</span></span>\n</span><span class=\"syntax-punctuation syntax-section syntax-entry syntax-end syntax-bibtex\">}</span></span>\n</span></span></code></pre>\n<hr />\n<p><strong>Try BigCodeArena now</strong>: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://huggingface.co/spaces/bigcode/arena\">Hugging Face Space</a></p>\n<p><strong>Read the paper</strong>: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://huggingface.co/papers/2510.08697\">Hugging Face</a></p>\n<p><strong>Run the code</strong>: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://github.com/bigcode-project/bigcodearena\">GitHub</a></p>\n<p><strong>Explore the collection</strong>: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://huggingface.co/collections/bigcode/bigcodearena-68cd3a196e5147cc45f8ea3d\">Hugging Face Collection</a></p>\n",
  "content_markdown": "Evaluating the quality of AI-generated code is notoriously difficult. While humans can easily spot whether a piece of code \"looks right,\" determining if it actually works correctly, handles edge cases properly, and produces the intended result requires running and testing it. This is why today, we're thrilled to announce **BigCodeArena** -- the first human-in-the-loop platform for evaluating code generation models through execution.\n\nInspired by LMArena for LLMs, we've built a platform that allows anyone to compare code generation models side-by-side, but with a crucial difference: **you can actually run the code and see what it produces**. Just submit a coding task, watch two different models generate solutions, execute both programs, and vote on which model produced better results. The outcomes are organized into a leaderboard that displays the community's highest-rated models.\n\n## Motivation\n\nThe field of code generation has long struggled with reliable evaluation methods. Traditional benchmarks like HumanEval test code against predefined test cases, but these represent only a tiny fraction of real-world programming tasks. Human evaluation platforms exist for general chatbots, but they fall short for code: reading raw source code and mentally simulating its execution is cognitively demanding and error-prone, especially for longer programs or complex UI applications.\n\nConsider this scenario:\n\n> You ask two AI models to build a responsive photo gallery website. Both generate code that looks syntactically correct. But which one is actually better? Without running the code, it's nearly impossible to tell. One might produce a beautiful, functional grid layout, while the other might have subtle bugs or poor styling that only become apparent when rendered in a browser.\n\n[![bigcodearena-demo](https://cdn-uploads.huggingface.co/production/uploads/62b7fb545233925f253531c8/SspOHYTm-BT-zIlIV3Rdg.png)](https://cdn-uploads.huggingface.co/production/uploads/62b7fb545233925f253531c8/SspOHYTm-BT-zIlIV3Rdg.png)\n\nThis observation led us to a key insight: **execution feedback is essential for humans to judge code quality reliably.** That's exactly what BigCodeArena provides.\n\n## The BigCodeArena Platform\n\nBigCodeArena extends the Chatbot Arena framework with powerful features specifically designed for code evaluation:\n\n### Real-Time Execution\n\nEvery code snippet generated by models is automatically executed in isolated sandbox environments. Whether it's a Python script, a React web app, a PyGame game, or a C++ algorithm, you can see the actual output, not just the source code.\n\n### Multi-Language & Framework Support\n\nWe currently support 10 languages (Python, JavaScript, TypeScript, HTML, C, C++, Java, Go, Rust, and Markdown) and 8 execution environments:\n\n- **Web Frameworks**: React, Vue, Core Web (vanilla HTML/CSS/JS)\n- **Python Frameworks**: Streamlit, Gradio, PyGame\n- **Diagrams**: Mermaid\n- **General Purpose Interpreters**: Python and JavaScript code interpreters, plus compiled language runners\n\n### Interactive Testing\n\nUnlike static code comparison, you can actually interact with the generated applications:\n\n- Click buttons and test UI elements in web apps\n- Play the games generated by models\n- Edit the code and re-run it to test modifications\n- View visual outputs like plots, charts, and diagrams\n\n[![bigcodearena](https://cdn-uploads.huggingface.co/production/uploads/62b7fb545233925f253531c8/2AvLXvIP1VE1moRw9BHS_.png)](https://cdn-uploads.huggingface.co/production/uploads/62b7fb545233925f253531c8/2AvLXvIP1VE1moRw9BHS_.png)\n\n### Multi-Turn Conversations\n\nReal programming isn't one-and-done. BigCodeArena supports multi-turn interactions, allowing you to refine requirements, ask for features to be added, or request bug fixes -- just like working with a real coding assistant.\n\n## What We've Learned: 5 Months of Community Evaluation\n\nSince launching in February 2025, BigCodeArena has collected over **14,000 conversations** from more than 500 unique users, with **4,700+ high-quality preference votes** comparing 10 frontier LLMs.\n\n### Programming Topics in the Wild\n\nOur users have explored remarkably diverse coding scenarios:\n\n- **Web Design (36%)**: Building responsive websites, interactive dashboards, and web applications\n- **Problem Solving (23%)**: Algorithms, data structures, and computational challenges\n- **Game Development (16%)**: Creating interactive games with physics, collision detection, and graphics\n- **Scientific Computing (14%)**: Data analysis, visualization, and numerical simulations\n- **Creative Coding (8%)**: Artistic visualizations, generative art, and experimental interfaces\n- **Diagram Creation (3%)**: Flowcharts, system architectures, and data visualizations\n\n### Language and Framework Popularity\n\nPython dominates with over 4,000 conversations, followed by JavaScript/TypeScript (3,359), HTML (1,601), and C++ (642). Among frameworks, direct Python interpreters lead usage (6,000 sessions), with React (2,729), Core Web (1,574), Streamlit (1,254), and PyGame (1,087) also seeing heavy use.\n\n### User Interaction Patterns\n\nMost interactions are focused and efficient: 76% of conversations consist of just 2 turns (one request, one response), with a mean conversation length of 4.12 messages. However, the platform supports extended multi-turn debugging sessions when needed, with some conversations exceeding 10 turns as users refine complex applications.\n\n### Model Rankings from Community Votes\n\nFrom our 14K conversations, we filtered for high-quality pairwise comparisons: conversations with at least two turns and actual code execution. This yielded **4,731 voting samples**, with each evaluated model receiving at least 700 votes. We aggregate these votes into **Elo ratings** using the Bradley-Terry model, which estimates the probability that one model beats another based on head-to-head comparisons.\n\nTo ensure robust rankings, we use 100 bootstrap resamples to construct 95% confidence intervals, so we can identify statistically significant performance differences between models.\n\n[![elo_ratings_comparison_page-0001](https://cdn-uploads.huggingface.co/production/uploads/62b7fb545233925f253531c8/rEnROVN53CxyMHsLh223J.jpeg)](https://cdn-uploads.huggingface.co/production/uploads/62b7fb545233925f253531c8/rEnROVN53CxyMHsLh223J.jpeg)\n\nWe evaluate models under three settings to control for different factors:\n\n1. **All Data**: Uses all pairwise comparisons regardless of execution environment or programming language\n2. **Environment Matched**: Only compares models when both were executed in the same sandbox (e.g., both in React or both in PyGame)\n3. **Language Matched**: Further restricts comparisons to the same programming language\n\nRankings remain remarkably consistent across all three settings, revealing clear performance tiers:\n\n**Top Tier**: **o3-mini** and **o1-mini** consistently lead with the highest Elo ratings and tight confidence intervals. These models maintain top performance regardless of environment or language constraints, showing strong robustness across coding scenarios. **Claude-3.5-Sonnet** follows closely, particularly excelling when language is controlled.\n\n**Mid Tier**: **GPT-4o**, **o1**, and **Gemini-2.0-Pro/Flash** form a competitive middle tier. GPT-4o shows some sensitivity to language matching, suggesting room for improvement in multilingual consistency.\n\n**Open Source Models**: **Qwen2.5** variants and **Llama-3.3-70B** lag behind frontier proprietary models, highlighting the performance gap that remains between leading closed and open models.\n\n[![combined_performance_heatmaps_page-0001](https://cdn-uploads.huggingface.co/production/uploads/62b7fb545233925f253531c8/5iB4XRwOBzjEdQM5tqVoU.jpeg)](https://cdn-uploads.huggingface.co/production/uploads/62b7fb545233925f253531c8/5iB4XRwOBzjEdQM5tqVoU.jpeg)\n\n*Figure: Overall win rate heatmaps (percentage of all pairwise comparisons won) of each model in the sessions across languages (left) and execution environments (right). For each category, we only keep models that appear in at least 3 conversation sessions.*\n\n#### Performance Across Languages\n\nBreaking down performance by programming language reveals interesting patterns:\n\n- Top-tier models like **o3-mini** and **o1-mini** achieve dominant win rates in mainstream languages like Python, Java, and C++\n- **Gemini-2.0-Pro** shows particular strength in Rust, achieving the highest win rate in that category\n- Different models exhibit distinct areas of expertise, with frontier models excelling in different niches\n- Open models like Qwen2.5 variants show inconsistent performance, particularly struggling with Rust and Go\n\n#### Performance Across Execution Environments\n\nAnalyzing win rates by execution environment reveals how models handle different runtime contexts:\n\n**Robust Performers**: **o3-mini** maintains consistently strong performance across React, Streamlit, Gradio, Core Web, and PyGame, demonstrating excellent environmental adaptability.\n\n**Stable but Selective**: **Claude-3.5-Sonnet** and **Gemini-2.0-Flash** show generally stable performance but with reduced win rates in complex UI-heavy environments like Vue and Mermaid.\n\n**Framework-Specific Weaknesses**: **Qwen2.5** models, while competitive in some web frameworks (Core Web, React), struggle significantly with interactive and visualization-oriented environments like PyGame, Vue, and Mermaid. These environments often require precise handling of control flow, graphics rendering, and package dependencies.\n\nThese results highlight an important insight: **aggregate Elo scores don't tell the whole story**. Some models remain brittle under specific runtime constraints, and execution environment matters significantly for real-world deployment.\n\n## Two New Benchmarks: BigCodeReward and AutoCodeArena\n\nTo advance research beyond crowdsourced evaluation, we're releasing two complementary benchmarks:\n\n### BigCodeReward: Evaluating Reward Models for Code\n\nBuilding on our 4,700+ preference votes, BigCodeReward tests how well LLMs can judge code quality when acting as reward models. The key finding? **Execution results dramatically improve judgment accuracy.**\n\nWhen models can see execution outputs (screenshots of web apps, game visuals, program logs), their alignment with human preferences increases substantially:\n\n- Claude-Sonnet-4: **56.7% → 62.3%** accuracy\n- GPT-4o: **54.6% → 63.8%** accuracy\n- Qwen2.5-VL-72B: **58.7% → 66.2%** accuracy\n\nThis reinforces our core thesis: you can't reliably judge code without running it -- and this applies to both humans *and* AI judges.\n\n### AutoCodeArena: Automated Code Generation Benchmarks\n\nInspired by Arena-Hard-Auto, AutoCodeArena provides a scalable way to evaluate new models without waiting for thousands of human votes. We carefully selected 600 representative prompts from our crowdsourced data, spanning all programming topics and frameworks.\n\nUsing automated LLM judges (Claude-3.7-Sonnet) to evaluate code execution results against a GPT-4.1 baseline, we can rapidly benchmark new models. This approach enables weekly leaderboard updates as new models are released.\n\nOur automated benchmark evaluated 20+ cutting-edge models, including recently released systems:\n\n**Top Performers:**\n\n1. **GPT-5** -- Establishes new state-of-the-art by a significant margin\n2. **Claude-Opus-4** and **Claude-Sonnet-4** -- Strong second tier, excelling in reasoning-heavy tasks\n3. **Qwen3-Coder**, **Kimi-K2**, **GLM-4.5** -- Leading open models that narrow the gap with mid-tier proprietary systems\n\n[![overall_performance](https://cdn-uploads.huggingface.co/production/uploads/62b7fb545233925f253531c8/NzyobCl-pYwgR-6ggufMe.jpeg)](https://cdn-uploads.huggingface.co/production/uploads/62b7fb545233925f253531c8/NzyobCl-pYwgR-6ggufMe.jpeg)\n\n*Figure: Win rates of recent LLMs on AutoCodeArena against a GPT-4.1 baseline, judged by Claude-3.7-Sonnet. The 50% mark represents parity with GPT-4.1. Models above this line outperform the baseline, while those below underperform. Error bars show 95% confidence intervals. Note: Claude-3.7-Sonnet is excluded from rankings to avoid self-judgment bias, and GPT-4.1 appears only as the reference baseline.*\n\nThe results show that while proprietary models maintain an edge, open-source models are rapidly closing the gap, with some approaching GPT-4.1-level performance.\n\n## Try It Yourself\n\nBigCodeArena is open to everyone -- no account required! Visit [https://huggingface.co/spaces/bigcode/arena](https://huggingface.co/spaces/bigcode/arena) to:\n\n- Compare code from more recent frontier LLMs (e.g., Qwen3, DeepSeek-V3.X, and other proprietary models)\n- Test web apps, games, visualizations, and algorithms\n- See real execution results, not just source code\n- Vote on your preferences to help improve the leaderboard\n- Explore multi-turn coding conversations\n\nWhether you're building a React dashboard, creating a PyGame game, solving algorithmic challenges, or generating creative visualizations, BigCodeArena lets you see which models truly deliver.\n\n## Open Source Everything\n\nFollowing the BigCode Project's commitment to transparency, we're releasing:\n\n- **Codebase**: Full evaluation pipelines and Gradio application source ([GitHub](https://github.com/bigcode-project/bigcodearena))\n- **Crowdsourced Data**: 14K raw conversations and 4.7K preference votes ([HuggingFace Collection](https://huggingface.co/collections/bigcode/bigcodearena-68cd3a196e5147cc45f8ea3d))\n- **Benchmarks**: BigCodeReward and AutoCodeArena datasets\n\n## What's Next?\n\nWe envision BigCodeArena as a long-term project that evolves with the community:\n\n- **Expanded Language Support**: More programming languages and frameworks.\n- **Live Benchmarks**: Continuously refreshed evaluation prompts to prevent overfitting\n- **Agent-Based Evaluation**: Using AI agents to interact with web apps for deeper testing\n- **Better Reward Models**: Advancing automated code quality assessment\n- **Community Contributions**: We welcome new execution environments, evaluation criteria, and model additions. [PRs](https://huggingface.co/spaces/bigcode/arena/discussions?new_pr=true) are always welcome!\n\n## Conclusion\n\nEvaluating code isn't like evaluating text -- you need to run it, test it, and interact with it. BigCodeArena makes this possible at scale, combining human judgment with real execution feedback to create the most reliable evaluation platform for code generation models.\n\nJoin us in building the future of code generation evaluation. Write a prompt, compare the models, and vote for your favorite. Your feedback helps the entire community understand which models truly deliver on the promise of AI-assisted programming.\n\nWe'd love to hear your feedback! Connect with us on [GitHub](https://github.com/bigcode-project/bigcodearena), join discussions in the [Hugging Face Space community tab](https://huggingface.co/spaces/bigcode/arena/discussions), or reach out to the BigCode Project at `contact@bigcode-project.org`.\n\n## Acknowledgements\n\nWe thank [Leandro von Werra](https://huggingface.co/lvwerra) for his valuable suggestions and feedback on the blog.\n\n## Citation\n\n```bibtex\n@article{zhuo2025bigcodearena,\n    title={BigCodeArena: Unveiling More Reliable Human Preferences in Code Generation via Execution},\n    author={Terry Yue Zhuo, Xiaolong Jin, Hange Liu, Juyong Jiang, Tianyang Liu, Chen Gong, Bhupesh Bishnoi, Vaisakhi Mishra, Marek Suppa, Noah Ziems, Saiteja Utpala, Ming Xu, Guangyu Song, Kaixin Li, Yuhan Cao, Bo Liu, Zheng Liu, Sabina Abdurakhmanova, Wenhao Yu, Mengzhao Jia, Jihan Yao, Kenneth Hamilton, Kumar Shridhar, Minh Chien Vu, Dingmin Wang, Jiawei Liu, Zijian Wang, Qian Liu, Binyuan Hui, Meg Risdal, Ahsen Khaliq, Atin Sood, Zhenchang Xing, Wasi Uddin Ahmad, John Grundy, David Lo, Banghua Zhu, Xiaoning Du, Torsten Scholak, Leandro von Werra},\n    year={2025}\n}\n```\n\n* * *\n\n**Try BigCodeArena now**: [Hugging Face Space](https://huggingface.co/spaces/bigcode/arena)\n\n**Read the paper**: [Hugging Face](https://huggingface.co/papers/2510.08697)\n\n**Run the code**: [GitHub](https://github.com/bigcode-project/bigcodearena)\n\n**Explore the collection**: [Hugging Face Collection](https://huggingface.co/collections/bigcode/bigcodearena-68cd3a196e5147cc45f8ea3d)\n",
  "dateCreated": "2026-10-04T12:50:30.804479567+00:00",
  "date_created": "2025-10-07T09:37:25+00:00",
  "date_modified": "2025-10-07T09:37:25+00:00",
  "date_published": "2025-10-07T09:37:25+00:00",
  "description": "Evaluating the quality of AI-generated code is notoriously difficult. While humans can easily spot whether a piece of code \"looks right,\" determining if it actually works correctly, handles edge cases properly, and produces the intended…",
  "external_url": "https://huggingface.co/blog/bigcode/arena",
  "git": {
    "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e",
    "path": "items/huggingface-co/2025/10/2025-10-07-bigcodearena-judging-code-generations-end-to-end-with-code.md",
    "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/huggingface-co/2025/10/2025-10-07-bigcodearena-judging-code-generations-end-to-end-with-code.md"
  },
  "id": "urn:aggr:item:fde6f379d292f5c1ecd9d448d5244817bafabe38",
  "isBasedOn": {
    "@id": "https://huggingface.co/blog/bigcode/arena"
  },
  "mainEntity": {
    "@id": "https://huggingface.co/blog/bigcode/arena",
    "@type": "CreativeWork",
    "archivedAt": {
      "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2025-10-07-bigcodearena-judging-code-generations-end-to-end-with-code/#webpage"
    },
    "dateModified": "2025-10-07T09:37:25+00:00",
    "datePublished": "2025-10-07T09:37:25+00:00",
    "headline": "BigCodeArena: Judging code generations end to end with code executions",
    "timeRequired": "PT9M",
    "url": "https://huggingface.co/blog/bigcode/arena",
    "wordCount": 1991
  },
  "name": "Archived snapshot: BigCodeArena: Judging code generations end to end with code executions",
  "preview": null,
  "reading_minutes": 9,
  "replicated_at": null,
  "source": "huggingface-co",
  "source_name": "Hugging Face - Blog",
  "summary": null,
  "tags": [],
  "temporalCoverage": "2026-10-04T12:50:30.804479567+00:00",
  "title": "BigCodeArena: Judging code generations end to end with code executions",
  "url": "https://aggr.aymericbeaumet.com/items/huggingface-co/2025-10-07-bigcodearena-judging-code-generations-end-to-end-with-code/",
  "word_count": 1991
}