{ "@context": "https://schema.org", "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-04-21-qimma-qimw-mountain-a-quality-first-arabic-llm-leaderboard/#webpage", "@type": [ "WebPage", "ArchiveComponent" ], "articleBody": "[![image](https://cdn-uploads.huggingface.co/production/uploads/66c8620a79b42e5c941b0265/NCnP1M7Ce__41kiS-V7hW.png)](https://cdn-uploads.huggingface.co/production/uploads/66c8620a79b42e5c941b0265/NCnP1M7Ce__41kiS-V7hW.png)\n\n> **QIMMA validates benchmarks before evaluating models, ensuring reported scores reflect genuine Arabic language capability in LLMs.**\n\nIf you've been tracking Arabic LLM evaluation, you've probably noticed a growing tension: the number of benchmarks and leaderboards is expanding rapidly, but **are we actually measuring what we think we're measuring?**\n\nWe built **QIMMA** ู‚ู…ู‘ุฉ (Arabic for \"summit\"), to answer that question systematically. Instead of aggregating existing Arabic benchmarks as-is and running models on them, we applied a rigorous quality validation pipeline *before* any evaluation took place. What we found was sobering: even widely-used, well-regarded Arabic benchmarks contain systematic quality issues that can quietly corrupt evaluation results.\n\nThis post walks through what QIMMA is, how we built it, what problems we found, and what the model rankings look like once you clean things up.\n\n[![image](https://cdn-uploads.huggingface.co/production/uploads/66c8620a79b42e5c941b0265/JFGUze00t2twAEsj6XxCm.png)](https://cdn-uploads.huggingface.co/production/uploads/66c8620a79b42e5c941b0265/JFGUze00t2twAEsj6XxCm.png)\n\n* * *\n\n## ๐Ÿ” The Problem: Arabic NLP Evaluation Is Fragmented and Unvalidated\n\nArabic is spoken by over 400 million people across diverse dialects and cultural contexts, yet the Arabic NLP evaluation landscape remains fragmented. A few key pain points have motivated this work:\n\n**Translation issues.** Many Arabic benchmarks are translations from English. This introduces distributional shifts. Questions that feel natural in English become awkward or culturally misaligned in Arabic, making benchmark data less representative of how Arabic is naturally used.\n\n**Absent quality validation.** Even *native* Arabic benchmarks are often released without rigorous quality checks. Annotation inconsistencies, incorrect gold answers, encoding errors, and cultural bias in ground-truth labels have all been documented in established resources.\n\n**Reproducibility gaps.** Evaluation scripts and per-sample outputs are rarely released publicly, making it hard to audit results or build on prior work.\n\n**Coverage fragmentation.** Existing leaderboards cover isolated tasks and narrow domains, making holistic model assessment difficult.\n\nTo illustrate where QIMMA sits relative to existing platforms:\n\n| Leaderboard | Open Source | Native Arabic | Quality Validation | Coding Eval | Public Outputs |\n| ----------- | ----------- | ------------- | ------------------ | ----------- | -------------- |\n| OALL v1 | โœ… | Mixed | โŒ | โŒ | โœ… |\n| OALL v2 | โœ… | Mostly | โŒ | โŒ | โœ… |\n| BALSAM | Partial | 50% | โŒ | โŒ | โŒ |\n| AraGen | โœ… | 100% | โœ… | โŒ | โŒ |\n| SILMA ABL | โœ… | 100% | โœ… | โŒ | โœ… |\n| ILMAAM | Partial | 100% | โœ… | โŒ | โŒ |\n| HELM Arabic | โœ… | Mixed | โŒ | โŒ | โœ… |\n| โ›ฐ QIMMA | โœ… | 99% | โœ… | โœ… | โœ… |\n\nQIMMA is the only platform combining all five properties: open source, predominantly native Arabic content, systematic quality validation, code evaluation, and public per-sample inference outputs.\n\n* * *\n\n## โ›ฐ What's in QIMMA?\n\nQIMMA consolidates **109 subsets** from **14 source benchmarks** into a unified evaluation suite of over **52,000 samples**, spanning 7 domains:\n\n| Domain | Benchmarks | Task Types |\n| ----------------------- | ----------------------------------- | ---------- |\n| **Cultural** | AraDiCE-Culture, ArabCulture, PalmX | MCQ |\n| **STEM** | ArabicMMLU, GAT, 3LM STEM | MCQ |\n| **Legal** | ArabLegalQA, MizanQA | MCQ, QA |\n| **Medical** | MedArabiQ, MedAraBench | MCQ, QA |\n| **Safety** | AraTrust | MCQ |\n| **Poetry & Literature** | FannOrFlop | QA |\n| **Coding** | 3LM HumanEval+, 3LM MBPP+ | Code |\n\nA few things stand out about this design:\n\n- **99% native Arabic content.** The only exception is code evaluation, which is inherently language-agnostic.\n- **First Arabic leaderboard with code evaluation.** QIMMA integrates Arabic-adapted versions of HumanEval+ and MBPP+, making it possible to assess coding capability with Arabic-language problem statements.\n- **Diversity in Domains and Tasks.** QIMMA evaluates real-world competency areas including education, governance, healthcare, creative expression, and software development.\n\n* * *\n\n## ๐Ÿ”ฌ The Quality Validation Pipeline\n\nThis is the methodological heart of QIMMA. Before running a single model, we applied a **multi-stage validation pipeline** to every sample in every benchmark.\n\n### Stage 1: Multi-Model Automated Assessment\n\nEach sample was independently evaluated by two state-of-the-art LLMs:\n\n- **Qwen3-235B-A22B-Instruct**\n- **DeepSeek-V3-671B**\n\nWe chose two models with strong Arabic capability but different training data compositions, so that their *combined* judgment is more robust than either alone.\n\nEach model scores a sample against a **10-point rubric**, with binary scores (0 or 1) per criterion:\n\n![QIMMA pipeline](https://cdn-uploads.huggingface.co/production/uploads/66c8620a79b42e5c941b0265/KkWnEuAunRTalBhMkPQ02.png)\n\nA sample is eliminated if either model scores it below 7/10. Samples where both models agree on elimination are dropped immediately. However, where only one model flags a sample, it proceeds to human review in Stage 2.\n\n### Stage 2: Human Annotation and Review\n\nFlagged samples are reviewed by **native Arabic speakers** with cultural and dialectal familiarity. Human annotators make final calls on:\n\n- Cultural context and regional variation\n- Dialectal nuance\n- Subjective interpretation\n- Subtle quality issues automated assessment may miss\n\nFor culturally sensitive content, multiple perspectives are considered, since \"correctness\" can genuinely vary across Arab regions.\n\n* * *\n\n## โš ๏ธ What We Found: Systematic Quality Problems\n\nThe pipeline revealed recurring quality issues across benchmarks; not isolated errors, but **systematic patterns** reflecting gaps in how benchmarks were originally constructed.\n\n### By the Numbers\n\n| Benchmark | Total Samples | Discarded | Discard Rate |\n| --------------- | ------------- | --------- | ------------ |\n| ArabicMMLU | 14,163 | 436 | 3.1% |\n| MizanQA | 1,769 | 41 | 2.3% |\n| PalmX | 3,001 | 25 | 0.8% |\n| MedAraBench | 4,960 | 33 | 0.7% |\n| FannOrFlop | 6,984 | 43 | 0.6% |\n| ArabCulture | 3,482 | 7 | 0.2% |\n| MedArabiQ | 499 | 1 | 0.2% |\n| GAT | 13,986 | 1 | \\~0.0% |\n| 3LM STEM | 2,609 | 1 | \\~0.0% |\n| AraDiCE-Culture | 180 | 0 | 0.0% |\n| ArabLegalQA | 79 | 0 | 0.0% |\n| AraTrust | 522 | 0 | 0.0% |\n\n### Taxonomy of Issues Found\n\nโš–๏ธ Answer Quality\n\nFalse or mismatched gold indices, factually wrong answers, missing or raw text answers.\n\n๐Ÿ“„ Text & Formatting Quality\n\nCorrupt or illegible text, spelling and grammar errors, and duplicate samples.\n\n๐Ÿ’ฌ Cultural Sensitivity\n\nStereotype reinforcement and monolithic generalizations about diverse communities.\n\n๐Ÿค Gold Answer Compliance\n\nMisalignment of gold answers with evaluation protocols.\n\n* * *\n\n## ๐Ÿ’ป Code Benchmark: A Different Kind of Quality Work\n\nCode benchmarks required a different intervention. Rather than discarding samples, we **refined the Arabic problem statements** in 3LM's Arabic adaptations of HumanEval+ and MBPP+, leaving task identifiers, reference solutions, and test suites completely unchanged.\n\nThe modification rates were striking:\n\n| Benchmark | Total Prompts | Modified | Unchanged | Modification Rate |\n| -------------- | ------------- | -------- | --------- | ----------------- |\n| 3LM HumanEval+ | 164 | 145 | 19 | **88%** |\n| 3LM MBPP+ | 378 | 308 | 70 | **81%** |\n\nModifications fell into five categories:\n\n1. **Linguistic refinement** : normalizing toward natural Modern Standard Arabic and consistent imperative style\n2. **Clarity improvements** : fixing ambiguous instructions and unclear constraints\n3. **Consistency normalization** : standardizing mathematical terminology, punctuation, and example formatting\n4. **Structural corrections** : fixing broken triple-quoted strings, indentation errors, corrupted text fragments\n5. **Semantic refinements** : clarifying whether ranges are inclusive/exclusive, preserving task intent\n\n* * *\n\n## โš™๏ธ Evaluation Setup\n\n### Evaluation Framework\n\nQIMMA uses [LightEval](https://github.com/huggingface/lighteval), [EvalPlus](https://github.com/evalplus/evalplus) and [FannOrFlop](https://github.com/mbzuai-oryx/FannOrFlop) as its evaluation framework, chosen for consistency, multilingual community adoption, and reproducibility.\n\n### Metrics by Task Type\n\n| Task Type | Metric | Benchmarks |\n| -------------------- | ---------------------------------- | ------------------------------------------------------------------------------------------------ |\n| **MCQ** | Normalized Log-Likelihood Accuracy | AraDiCE-Culture, ArabicMMLU, ArabCulture, PalmX, 3LM STEM, MedArabiQ, GAT, MedAraBench, AraTrust |\n| **Multi-select MCQ** | Probability Mass on Gold Choices | MizanQA |\n| **Generative QA** | F1 BERTScore (AraBERT v02) | MedArabiQ, ArabLegalQA, FannOrFlop |\n| **Code** | Pass@1 | 3LM HumanEval+, 3LM MBPP+ |\n\n### Prompt Templates\n\nQIMMA standardizes prompting by question format, with six template types:\n\n![QIMMA prompt templates](https://cdn-uploads.huggingface.co/production/uploads/659bc8a7b0f43ed69f0b2300/C-WxGverw4w_pnf6IRUsB.png)\\\n **MCQ**: generic multiple choice ยท **MCQ-C**: multiple choice with context passage ยท **MCQ-I**: multiple choice with specific instructions (GAT analogy/completion) ยท **QA**: generic open-ended QA ยท **QA-C**: QA with context ยท **QA-F**: fill-in-the-blank QA\n\nAll prompts are in Arabic. For MizanQA and ArabCulture, benchmark-specific system prompts from the original papers are preserved.\n\n* * *\n\n## ๐Ÿ† Leaderboard Results\n\n*Results as of April 2026; covering top 10 evaluated models. Visit the [live leaderboard](https://huggingface.co/spaces/qimma/leaderboard) for current rankings.*\n\n| Rank | Model | AVERAGE | AraDiCE-Culture | ArabicMMLU | ArabCulture | PALMX | 3LM STEM | AraTrust | MizanQA | MedArabiQ | ArabLegalQA | GAT | MedAraBench | HumanEval+ | MBPP+ | FannOrFlop |\n| ---- | -------------------------------------- | --------- | --------------- | ---------- | ----------- | --------- | --------- | --------- | --------- | --------- | ----------- | --------- | ----------- | ---------- | --------- | ---------- |\n| ๐Ÿฅ‡ 1 | Qwen/Qwen3.5-397B-A17B-FP8 | **68.06** | **82.78** | 77.54 | 61.75 | **83.91** | 88.67 | 90.04 | **73.36** | 47.30 | 54.94 | 55.89 | 47.97 | **67.68** | **76.72** | 44.33 |\n| ๐Ÿฅˆ 2 | Applied-Innovation-Center/Karnak | 66.20 | 73.33 | 80.94 | 53.49 | 81.40 | **93.10** | 89.08 | 55.92 | 55.78 | **71.58** | 61.06 | 54.19 | 33.54 | 64.55 | 58.91 |\n| ๐Ÿฅ‰ 3 | inceptionai/Jais-2-70B-Chat | 65.81 | 78.89 | **81.29** | **83.24** | 83.73 | 87.96 | 90.23 | 71.78 | 52.79 | 69.60 | 51.67 | 50.89 | 19.51 | 43.65 | 56.13 |\n| #4 | Qwen/Qwen2.5-72B-Instruct | 65.75 | 77.22 | 73.78 | 63.83 | 77.77 | 87.55 | 88.51 | 63.49 | 50.06 | 70.74 | 55.90 | 44.19 | 37.20 | 72.75 | 57.51 |\n| #5 | Applied-Innovation-Center/AIC-1 | 65.37 | 73.33 | 72.02 | 77.52 | 76.11 | 88.13 | **90.61** | 56.36 | 53.75 | 68.96 | **62.11** | 50.78 | 28.05 | 69.58 | 47.83 |\n| #6 | Qwen/Qwen3.5-122B-A10B | 64.84 | 74.44 | 73.17 | 37.78 | 81.46 | 86.18 | 86.97 | 64.01 | 47.04 | 55.11 | 50.90 | 52.49 | 65.24 | 72.43 | 60.54 |\n| #7 | Sakalti/Ultiima-72B | 64.49 | 78.33 | 72.28 | 68.79 | 76.75 | 83.70 | 89.08 | 60.44 | 44.58 | 69.12 | 46.91 | 42.25 | 39.02 | 74.07 | 57.56 |\n| #8 | meta-llama/Llama-3.3-70B-Instruct | 63.96 | 77.22 | 71.57 | 78.05 | 77.95 | 88.28 | 85.63 | 67.44 | **56.25** | 64.00 | 51.13 | **54.86** | 27.44 | 71.16 | 24.43 |\n| #9 | Qwen/Qwen2.5-32B-Instruct | 63.26 | 70.56 | 68.76 | 75.80 | 72.07 | 81.03 | 85.82 | 53.78 | 48.08 | 69.27 | 56.94 | 36.51 | 34.15 | 72.75 | **93.10** |\n| #10 | FreedomIntelligence/AceGPT-v2-32B-Chat | 61.14 | 76.67 | 70.62 | 79.79 | 74.46 | 84.88 | 86.97 | 63.89 | 49.96 | 71.46 | 56.04 | 47.32 | 23.78 | 54.50 | 15.56 |\n\n- **Scale does not guarantee best performance.** The top 10 spans models from 32B to 397B parameters, with several mid-size models outperforming larger ones on specific domains.\n- **Arabic-specialized models lead on cultural and linguistic tasks.** Jais-2-70B-Chat ranks highest on ArabicMMLU and ArabCulture, while Karnak leads on 3LM STEM and ArabLegalQA.\n- **Coding remains the hardest domain for Arabic-specialized models.** The top HumanEval+ and MBPP+ scores belong to multilingual models, with Qwen3.5-397B leading both.\n\n### The Size-Performance Relationship\n\nAcross the full leaderboard (46 models), a clear but imperfect size-performance correlation emerges. However, there are interesting exceptions:\n\n[![c64aafc7-1](https://cdn-uploads.huggingface.co/production/uploads/66c8620a79b42e5c941b0265/KPdYTBwzMvJEyALTYfGji.png)](https://cdn-uploads.huggingface.co/production/uploads/66c8620a79b42e5c941b0265/KPdYTBwzMvJEyALTYfGji.png)\n\n- Arabic-specialized models often outperform size-matched multilingual models\n- Instruction-tuned models consistently outperform their base counterparts except for Qwen3\n- Some smaller Arabic-specialized models (Fanar-1-9B, ALLaM-7B) outperform much larger multilingual models on specific domains\n\n* * *\n\n## ๐ŸŒŸ What Makes QIMMA Different\n\nTo summarize the distinctive properties of QIMMA:\n\n| Property | Details |\n| ---------------------------- | ------------------------------------------------------------------------- |\n| **Quality-first philosophy** | Validation runs *before* evaluation, not as an afterthought |\n| **Multi-model validation** | Two LLMs with different training + human review for flagged cases |\n| **99% native Arabic** | Avoids translation artifacts almost entirely |\n| **Multi-domain, multi-task** | 7 domains, 3 task types (MCQ, QA, code), 109 subsets |\n| **Code evaluation** | First Arabic leaderboard to include code generation |\n| **Full transparency** | Per-sample inference outputs publicly released, not just aggregate scores |\n| **LightEval-based** | Unified, reproducible evaluation codebase |\n| **Dialectal awareness** | Explicit handling of MSA vs. dialectal variation in prompts and rubrics |\n\n* * *\n\n## ๐Ÿ”— Resources\n\n- ๐Ÿ† **Leaderboard**: [QIMMA Leaderboard](https://huggingface.co/spaces/qimma/leaderboard)\n- ๐Ÿ’ป **Code**: [GitHub](https://github.com/tiiuae/QIMMA-leaderboard.git)\n- ๐Ÿ“„ **Paper**: [Are Arabic Benchmarks Reliable? QIMMA's Quality-First Approach to LLM Evaluation](https://arxiv.org/pdf/2604.03395)\n\n* * *\n\n## ๐Ÿ”– Citation\n\n```bibtex\n@misc{alqadi2026arabicbenchmarksreliableqimmas,\n title={Are Arabic Benchmarks Reliable? QIMMA's Quality-First Approach to LLM Evaluation}, \n author={Leen AlQadi and Ahmed Alzubaidi and Mohammed Alyafeai and Hamza Alobeidli and Maitha Alhammadi and Shaikha Alsuwaidi and Omar Alkaabi and Basma El Amel Boussaha and Hakim Hacid},\n year={2026},\n eprint={2604.03395},\n archivePrefix={arXiv},\n primaryClass={cs.CL},\n url={https://arxiv.org/abs/2604.03395}, \n}\n```\n", "authors": [], "captured_at": "2026-10-19T01:26:53.998049939+00:00", "category": "ai", "content_html": "
\"image\"

\n
\n

QIMMA validates benchmarks before evaluating models, ensuring reported scores reflect genuine Arabic language capability in LLMs.

\n
\n

If you've been tracking Arabic LLM evaluation, you've probably noticed a growing tension: the number of benchmarks and leaderboards is expanding rapidly, but are we actually measuring what we think we're measuring?

\n

We built QIMMA ู‚ู…ู‘ุฉ (Arabic for "summit"), to answer that question systematically. Instead of aggregating existing Arabic benchmarks as-is and running models on them, we applied a rigorous quality validation pipeline before any evaluation took place. What we found was sobering: even widely-used, well-regarded Arabic benchmarks contain systematic quality issues that can quietly corrupt evaluation results.

\n

This post walks through what QIMMA is, how we built it, what problems we found, and what the model rankings look like once you clean things up.

\n

\"image\"

\n
\n

๐Ÿ” The Problem: Arabic NLP Evaluation Is Fragmented and Unvalidated

\n

Arabic is spoken by over 400 million people across diverse dialects and cultural contexts, yet the Arabic NLP evaluation landscape remains fragmented. A few key pain points have motivated this work:

\n

Translation issues. Many Arabic benchmarks are translations from English. This introduces distributional shifts. Questions that feel natural in English become awkward or culturally misaligned in Arabic, making benchmark data less representative of how Arabic is naturally used.

\n

Absent quality validation. Even native Arabic benchmarks are often released without rigorous quality checks. Annotation inconsistencies, incorrect gold answers, encoding errors, and cultural bias in ground-truth labels have all been documented in established resources.

\n

Reproducibility gaps. Evaluation scripts and per-sample outputs are rarely released publicly, making it hard to audit results or build on prior work.

\n

Coverage fragmentation. Existing leaderboards cover isolated tasks and narrow domains, making holistic model assessment difficult.

\n

To illustrate where QIMMA sits relative to existing platforms:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
LeaderboardOpen SourceNative ArabicQuality ValidationCoding EvalPublic Outputs
OALL v1โœ…MixedโŒโŒโœ…
OALL v2โœ…MostlyโŒโŒโœ…
BALSAMPartial50%โŒโŒโŒ
AraGenโœ…100%โœ…โŒโŒ
SILMA ABLโœ…100%โœ…โŒโœ…
ILMAAMPartial100%โœ…โŒโŒ
HELM Arabicโœ…MixedโŒโŒโœ…
โ›ฐ QIMMAโœ…99%โœ…โœ…โœ…
\n

QIMMA is the only platform combining all five properties: open source, predominantly native Arabic content, systematic quality validation, code evaluation, and public per-sample inference outputs.

\n
\n

โ›ฐ What's in QIMMA?

\n

QIMMA consolidates 109 subsets from 14 source benchmarks into a unified evaluation suite of over 52,000 samples, spanning 7 domains:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
DomainBenchmarksTask Types
CulturalAraDiCE-Culture, ArabCulture, PalmXMCQ
STEMArabicMMLU, GAT, 3LM STEMMCQ
LegalArabLegalQA, MizanQAMCQ, QA
MedicalMedArabiQ, MedAraBenchMCQ, QA
SafetyAraTrustMCQ
Poetry & LiteratureFannOrFlopQA
Coding3LM HumanEval+, 3LM MBPP+Code
\n

A few things stand out about this design:

\n\n
\n

๐Ÿ”ฌ The Quality Validation Pipeline

\n

This is the methodological heart of QIMMA. Before running a single model, we applied a multi-stage validation pipeline to every sample in every benchmark.

\n

Stage 1: Multi-Model Automated Assessment

\n

Each sample was independently evaluated by two state-of-the-art LLMs:

\n\n

We chose two models with strong Arabic capability but different training data compositions, so that their combined judgment is more robust than either alone.

\n

Each model scores a sample against a 10-point rubric, with binary scores (0 or 1) per criterion:

\n

\"QIMMA

\n

A sample is eliminated if either model scores it below 7/10. Samples where both models agree on elimination are dropped immediately. However, where only one model flags a sample, it proceeds to human review in Stage 2.

\n

Stage 2: Human Annotation and Review

\n

Flagged samples are reviewed by native Arabic speakers with cultural and dialectal familiarity. Human annotators make final calls on:

\n\n

For culturally sensitive content, multiple perspectives are considered, since "correctness" can genuinely vary across Arab regions.

\n
\n

โš ๏ธ What We Found: Systematic Quality Problems

\n

The pipeline revealed recurring quality issues across benchmarks; not isolated errors, but systematic patterns reflecting gaps in how benchmarks were originally constructed.

\n

By the Numbers

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
BenchmarkTotal SamplesDiscardedDiscard Rate
ArabicMMLU14,1634363.1%
MizanQA1,769412.3%
PalmX3,001250.8%
MedAraBench4,960330.7%
FannOrFlop6,984430.6%
ArabCulture3,48270.2%
MedArabiQ49910.2%
GAT13,9861~0.0%
3LM STEM2,6091~0.0%
AraDiCE-Culture18000.0%
ArabLegalQA7900.0%
AraTrust52200.0%
\n

Taxonomy of Issues Found

\n

โš–๏ธ Answer Quality

\n

False or mismatched gold indices, factually wrong answers, missing or raw text answers.

\n

๐Ÿ“„ Text & Formatting Quality

\n

Corrupt or illegible text, spelling and grammar errors, and duplicate samples.

\n

๐Ÿ’ฌ Cultural Sensitivity

\n

Stereotype reinforcement and monolithic generalizations about diverse communities.

\n

๐Ÿค Gold Answer Compliance

\n

Misalignment of gold answers with evaluation protocols.

\n
\n

๐Ÿ’ป Code Benchmark: A Different Kind of Quality Work

\n

Code benchmarks required a different intervention. Rather than discarding samples, we refined the Arabic problem statements in 3LM's Arabic adaptations of HumanEval+ and MBPP+, leaving task identifiers, reference solutions, and test suites completely unchanged.

\n

The modification rates were striking:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
BenchmarkTotal PromptsModifiedUnchangedModification Rate
3LM HumanEval+1641451988%
3LM MBPP+3783087081%
\n

Modifications fell into five categories:

\n
    \n
  1. Linguistic refinement : normalizing toward natural Modern Standard Arabic and consistent imperative style
    2. \n
  2. Clarity improvements : fixing ambiguous instructions and unclear constraints
    3. \n
  3. Consistency normalization : standardizing mathematical terminology, punctuation, and example formatting
    4. \n
  4. Structural corrections : fixing broken triple-quoted strings, indentation errors, corrupted text fragments
    5. \n
  5. Semantic refinements : clarifying whether ranges are inclusive/exclusive, preserving task intent
    6. \n
\n
\n

โš™๏ธ Evaluation Setup

\n

Evaluation Framework

\n

QIMMA uses LightEval, EvalPlus and FannOrFlop as its evaluation framework, chosen for consistency, multilingual community adoption, and reproducibility.

\n

Metrics by Task Type

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Task TypeMetricBenchmarks
MCQNormalized Log-Likelihood AccuracyAraDiCE-Culture, ArabicMMLU, ArabCulture, PalmX, 3LM STEM, MedArabiQ, GAT, MedAraBench, AraTrust
Multi-select MCQProbability Mass on Gold ChoicesMizanQA
Generative QAF1 BERTScore (AraBERT v02)MedArabiQ, ArabLegalQA, FannOrFlop
CodePass@13LM HumanEval+, 3LM MBPP+
\n

Prompt Templates

\n

QIMMA standardizes prompting by question format, with six template types:

\n

\"QIMMA

MCQ: generic multiple choice ยท MCQ-C: multiple choice with context passage ยท MCQ-I: multiple choice with specific instructions (GAT analogy/completion) ยท QA: generic open-ended QA ยท QA-C: QA with context ยท QA-F: fill-in-the-blank QA
\n

All prompts are in Arabic. For MizanQA and ArabCulture, benchmark-specific system prompts from the original papers are preserved.

\n
\n

๐Ÿ† Leaderboard Results

\n

Results as of April 2026; covering top 10 evaluated models. Visit the live leaderboard for current rankings.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
RankModelAVERAGEAraDiCE-CultureArabicMMLUArabCulturePALMX3LM STEMAraTrustMizanQAMedArabiQArabLegalQAGATMedAraBenchHumanEval+MBPP+FannOrFlop
๐Ÿฅ‡ 1Qwen/Qwen3.5-397B-A17B-FP868.0682.7877.5461.7583.9188.6790.0473.3647.3054.9455.8947.9767.6876.7244.33
๐Ÿฅˆ 2Applied-Innovation-Center/Karnak66.2073.3380.9453.4981.4093.1089.0855.9255.7871.5861.0654.1933.5464.5558.91
๐Ÿฅ‰ 3inceptionai/Jais-2-70B-Chat65.8178.8981.2983.2483.7387.9690.2371.7852.7969.6051.6750.8919.5143.6556.13
#4Qwen/Qwen2.5-72B-Instruct65.7577.2273.7863.8377.7787.5588.5163.4950.0670.7455.9044.1937.2072.7557.51
#5Applied-Innovation-Center/AIC-165.3773.3372.0277.5276.1188.1390.6156.3653.7568.9662.1150.7828.0569.5847.83
#6Qwen/Qwen3.5-122B-A10B64.8474.4473.1737.7881.4686.1886.9764.0147.0455.1150.9052.4965.2472.4360.54
#7Sakalti/Ultiima-72B64.4978.3372.2868.7976.7583.7089.0860.4444.5869.1246.9142.2539.0274.0757.56
#8meta-llama/Llama-3.3-70B-Instruct63.9677.2271.5778.0577.9588.2885.6367.4456.2564.0051.1354.8627.4471.1624.43
#9Qwen/Qwen2.5-32B-Instruct63.2670.5668.7675.8072.0781.0385.8253.7848.0869.2756.9436.5134.1572.7593.10
#10FreedomIntelligence/AceGPT-v2-32B-Chat61.1476.6770.6279.7974.4684.8886.9763.8949.9671.4656.0447.3223.7854.5015.56
\n\n

The Size-Performance Relationship

\n

Across the full leaderboard (46 models), a clear but imperfect size-performance correlation emerges. However, there are interesting exceptions:

\n

\"c64aafc7-1\"

\n\n
\n

๐ŸŒŸ What Makes QIMMA Different

\n

To summarize the distinctive properties of QIMMA:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
PropertyDetails
Quality-first philosophyValidation runs before evaluation, not as an afterthought
Multi-model validationTwo LLMs with different training + human review for flagged cases
99% native ArabicAvoids translation artifacts almost entirely
Multi-domain, multi-task7 domains, 3 task types (MCQ, QA, code), 109 subsets
Code evaluationFirst Arabic leaderboard to include code generation
Full transparencyPer-sample inference outputs publicly released, not just aggregate scores
LightEval-basedUnified, reproducible evaluation codebase
Dialectal awarenessExplicit handling of MSA vs. dialectal variation in prompts and rubrics
\n
\n

๐Ÿ”— Resources

\n\n
\n

๐Ÿ”– Citation

\n
@misc{alqadi2026arabicbenchmarksreliableqimmas,\n      title={Are Arabic Benchmarks Reliable? QIMMA's Quality-First Approach to LLM Evaluation}, \n      author={Leen AlQadi and Ahmed Alzubaidi and Mohammed Alyafeai and Hamza Alobeidli and Maitha Alhammadi and Shaikha Alsuwaidi and Omar Alkaabi and Basma El Amel Boussaha and Hakim Hacid},\n      year={2026},\n      eprint={2604.03395},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL},\n      url={https://arxiv.org/abs/2604.03395}, \n}\n
\n", "content_markdown": "[![image](https://cdn-uploads.huggingface.co/production/uploads/66c8620a79b42e5c941b0265/NCnP1M7Ce__41kiS-V7hW.png)](https://cdn-uploads.huggingface.co/production/uploads/66c8620a79b42e5c941b0265/NCnP1M7Ce__41kiS-V7hW.png)\n\n> **QIMMA validates benchmarks before evaluating models, ensuring reported scores reflect genuine Arabic language capability in LLMs.**\n\nIf you've been tracking Arabic LLM evaluation, you've probably noticed a growing tension: the number of benchmarks and leaderboards is expanding rapidly, but **are we actually measuring what we think we're measuring?**\n\nWe built **QIMMA** ู‚ู…ู‘ุฉ (Arabic for \"summit\"), to answer that question systematically. Instead of aggregating existing Arabic benchmarks as-is and running models on them, we applied a rigorous quality validation pipeline *before* any evaluation took place. What we found was sobering: even widely-used, well-regarded Arabic benchmarks contain systematic quality issues that can quietly corrupt evaluation results.\n\nThis post walks through what QIMMA is, how we built it, what problems we found, and what the model rankings look like once you clean things up.\n\n[![image](https://cdn-uploads.huggingface.co/production/uploads/66c8620a79b42e5c941b0265/JFGUze00t2twAEsj6XxCm.png)](https://cdn-uploads.huggingface.co/production/uploads/66c8620a79b42e5c941b0265/JFGUze00t2twAEsj6XxCm.png)\n\n* * *\n\n## ๐Ÿ” The Problem: Arabic NLP Evaluation Is Fragmented and Unvalidated\n\nArabic is spoken by over 400 million people across diverse dialects and cultural contexts, yet the Arabic NLP evaluation landscape remains fragmented. A few key pain points have motivated this work:\n\n**Translation issues.** Many Arabic benchmarks are translations from English. This introduces distributional shifts. Questions that feel natural in English become awkward or culturally misaligned in Arabic, making benchmark data less representative of how Arabic is naturally used.\n\n**Absent quality validation.** Even *native* Arabic benchmarks are often released without rigorous quality checks. Annotation inconsistencies, incorrect gold answers, encoding errors, and cultural bias in ground-truth labels have all been documented in established resources.\n\n**Reproducibility gaps.** Evaluation scripts and per-sample outputs are rarely released publicly, making it hard to audit results or build on prior work.\n\n**Coverage fragmentation.** Existing leaderboards cover isolated tasks and narrow domains, making holistic model assessment difficult.\n\nTo illustrate where QIMMA sits relative to existing platforms:\n\n| Leaderboard | Open Source | Native Arabic | Quality Validation | Coding Eval | Public Outputs |\n| ----------- | ----------- | ------------- | ------------------ | ----------- | -------------- |\n| OALL v1 | โœ… | Mixed | โŒ | โŒ | โœ… |\n| OALL v2 | โœ… | Mostly | โŒ | โŒ | โœ… |\n| BALSAM | Partial | 50% | โŒ | โŒ | โŒ |\n| AraGen | โœ… | 100% | โœ… | โŒ | โŒ |\n| SILMA ABL | โœ… | 100% | โœ… | โŒ | โœ… |\n| ILMAAM | Partial | 100% | โœ… | โŒ | โŒ |\n| HELM Arabic | โœ… | Mixed | โŒ | โŒ | โœ… |\n| โ›ฐ QIMMA | โœ… | 99% | โœ… | โœ… | โœ… |\n\nQIMMA is the only platform combining all five properties: open source, predominantly native Arabic content, systematic quality validation, code evaluation, and public per-sample inference outputs.\n\n* * *\n\n## โ›ฐ What's in QIMMA?\n\nQIMMA consolidates **109 subsets** from **14 source benchmarks** into a unified evaluation suite of over **52,000 samples**, spanning 7 domains:\n\n| Domain | Benchmarks | Task Types |\n| ----------------------- | ----------------------------------- | ---------- |\n| **Cultural** | AraDiCE-Culture, ArabCulture, PalmX | MCQ |\n| **STEM** | ArabicMMLU, GAT, 3LM STEM | MCQ |\n| **Legal** | ArabLegalQA, MizanQA | MCQ, QA |\n| **Medical** | MedArabiQ, MedAraBench | MCQ, QA |\n| **Safety** | AraTrust | MCQ |\n| **Poetry & Literature** | FannOrFlop | QA |\n| **Coding** | 3LM HumanEval+, 3LM MBPP+ | Code |\n\nA few things stand out about this design:\n\n- **99% native Arabic content.** The only exception is code evaluation, which is inherently language-agnostic.\n- **First Arabic leaderboard with code evaluation.** QIMMA integrates Arabic-adapted versions of HumanEval+ and MBPP+, making it possible to assess coding capability with Arabic-language problem statements.\n- **Diversity in Domains and Tasks.** QIMMA evaluates real-world competency areas including education, governance, healthcare, creative expression, and software development.\n\n* * *\n\n## ๐Ÿ”ฌ The Quality Validation Pipeline\n\nThis is the methodological heart of QIMMA. Before running a single model, we applied a **multi-stage validation pipeline** to every sample in every benchmark.\n\n### Stage 1: Multi-Model Automated Assessment\n\nEach sample was independently evaluated by two state-of-the-art LLMs:\n\n- **Qwen3-235B-A22B-Instruct**\n- **DeepSeek-V3-671B**\n\nWe chose two models with strong Arabic capability but different training data compositions, so that their *combined* judgment is more robust than either alone.\n\nEach model scores a sample against a **10-point rubric**, with binary scores (0 or 1) per criterion:\n\n![QIMMA pipeline](https://cdn-uploads.huggingface.co/production/uploads/66c8620a79b42e5c941b0265/KkWnEuAunRTalBhMkPQ02.png)\n\nA sample is eliminated if either model scores it below 7/10. Samples where both models agree on elimination are dropped immediately. However, where only one model flags a sample, it proceeds to human review in Stage 2.\n\n### Stage 2: Human Annotation and Review\n\nFlagged samples are reviewed by **native Arabic speakers** with cultural and dialectal familiarity. Human annotators make final calls on:\n\n- Cultural context and regional variation\n- Dialectal nuance\n- Subjective interpretation\n- Subtle quality issues automated assessment may miss\n\nFor culturally sensitive content, multiple perspectives are considered, since \"correctness\" can genuinely vary across Arab regions.\n\n* * *\n\n## โš ๏ธ What We Found: Systematic Quality Problems\n\nThe pipeline revealed recurring quality issues across benchmarks; not isolated errors, but **systematic patterns** reflecting gaps in how benchmarks were originally constructed.\n\n### By the Numbers\n\n| Benchmark | Total Samples | Discarded | Discard Rate |\n| --------------- | ------------- | --------- | ------------ |\n| ArabicMMLU | 14,163 | 436 | 3.1% |\n| MizanQA | 1,769 | 41 | 2.3% |\n| PalmX | 3,001 | 25 | 0.8% |\n| MedAraBench | 4,960 | 33 | 0.7% |\n| FannOrFlop | 6,984 | 43 | 0.6% |\n| ArabCulture | 3,482 | 7 | 0.2% |\n| MedArabiQ | 499 | 1 | 0.2% |\n| GAT | 13,986 | 1 | \\~0.0% |\n| 3LM STEM | 2,609 | 1 | \\~0.0% |\n| AraDiCE-Culture | 180 | 0 | 0.0% |\n| ArabLegalQA | 79 | 0 | 0.0% |\n| AraTrust | 522 | 0 | 0.0% |\n\n### Taxonomy of Issues Found\n\nโš–๏ธ Answer Quality\n\nFalse or mismatched gold indices, factually wrong answers, missing or raw text answers.\n\n๐Ÿ“„ Text & Formatting Quality\n\nCorrupt or illegible text, spelling and grammar errors, and duplicate samples.\n\n๐Ÿ’ฌ Cultural Sensitivity\n\nStereotype reinforcement and monolithic generalizations about diverse communities.\n\n๐Ÿค Gold Answer Compliance\n\nMisalignment of gold answers with evaluation protocols.\n\n* * *\n\n## ๐Ÿ’ป Code Benchmark: A Different Kind of Quality Work\n\nCode benchmarks required a different intervention. Rather than discarding samples, we **refined the Arabic problem statements** in 3LM's Arabic adaptations of HumanEval+ and MBPP+, leaving task identifiers, reference solutions, and test suites completely unchanged.\n\nThe modification rates were striking:\n\n| Benchmark | Total Prompts | Modified | Unchanged | Modification Rate |\n| -------------- | ------------- | -------- | --------- | ----------------- |\n| 3LM HumanEval+ | 164 | 145 | 19 | **88%** |\n| 3LM MBPP+ | 378 | 308 | 70 | **81%** |\n\nModifications fell into five categories:\n\n1. **Linguistic refinement** : normalizing toward natural Modern Standard Arabic and consistent imperative style\n2. **Clarity improvements** : fixing ambiguous instructions and unclear constraints\n3. **Consistency normalization** : standardizing mathematical terminology, punctuation, and example formatting\n4. **Structural corrections** : fixing broken triple-quoted strings, indentation errors, corrupted text fragments\n5. **Semantic refinements** : clarifying whether ranges are inclusive/exclusive, preserving task intent\n\n* * *\n\n## โš™๏ธ Evaluation Setup\n\n### Evaluation Framework\n\nQIMMA uses [LightEval](https://github.com/huggingface/lighteval), [EvalPlus](https://github.com/evalplus/evalplus) and [FannOrFlop](https://github.com/mbzuai-oryx/FannOrFlop) as its evaluation framework, chosen for consistency, multilingual community adoption, and reproducibility.\n\n### Metrics by Task Type\n\n| Task Type | Metric | Benchmarks |\n| -------------------- | ---------------------------------- | ------------------------------------------------------------------------------------------------ |\n| **MCQ** | Normalized Log-Likelihood Accuracy | AraDiCE-Culture, ArabicMMLU, ArabCulture, PalmX, 3LM STEM, MedArabiQ, GAT, MedAraBench, AraTrust |\n| **Multi-select MCQ** | Probability Mass on Gold Choices | MizanQA |\n| **Generative QA** | F1 BERTScore (AraBERT v02) | MedArabiQ, ArabLegalQA, FannOrFlop |\n| **Code** | Pass@1 | 3LM HumanEval+, 3LM MBPP+ |\n\n### Prompt Templates\n\nQIMMA standardizes prompting by question format, with six template types:\n\n![QIMMA prompt templates](https://cdn-uploads.huggingface.co/production/uploads/659bc8a7b0f43ed69f0b2300/C-WxGverw4w_pnf6IRUsB.png)\\\n **MCQ**: generic multiple choice ยท **MCQ-C**: multiple choice with context passage ยท **MCQ-I**: multiple choice with specific instructions (GAT analogy/completion) ยท **QA**: generic open-ended QA ยท **QA-C**: QA with context ยท **QA-F**: fill-in-the-blank QA\n\nAll prompts are in Arabic. For MizanQA and ArabCulture, benchmark-specific system prompts from the original papers are preserved.\n\n* * *\n\n## ๐Ÿ† Leaderboard Results\n\n*Results as of April 2026; covering top 10 evaluated models. Visit the [live leaderboard](https://huggingface.co/spaces/qimma/leaderboard) for current rankings.*\n\n| Rank | Model | AVERAGE | AraDiCE-Culture | ArabicMMLU | ArabCulture | PALMX | 3LM STEM | AraTrust | MizanQA | MedArabiQ | ArabLegalQA | GAT | MedAraBench | HumanEval+ | MBPP+ | FannOrFlop |\n| ---- | -------------------------------------- | --------- | --------------- | ---------- | ----------- | --------- | --------- | --------- | --------- | --------- | ----------- | --------- | ----------- | ---------- | --------- | ---------- |\n| ๐Ÿฅ‡ 1 | Qwen/Qwen3.5-397B-A17B-FP8 | **68.06** | **82.78** | 77.54 | 61.75 | **83.91** | 88.67 | 90.04 | **73.36** | 47.30 | 54.94 | 55.89 | 47.97 | **67.68** | **76.72** | 44.33 |\n| ๐Ÿฅˆ 2 | Applied-Innovation-Center/Karnak | 66.20 | 73.33 | 80.94 | 53.49 | 81.40 | **93.10** | 89.08 | 55.92 | 55.78 | **71.58** | 61.06 | 54.19 | 33.54 | 64.55 | 58.91 |\n| ๐Ÿฅ‰ 3 | inceptionai/Jais-2-70B-Chat | 65.81 | 78.89 | **81.29** | **83.24** | 83.73 | 87.96 | 90.23 | 71.78 | 52.79 | 69.60 | 51.67 | 50.89 | 19.51 | 43.65 | 56.13 |\n| #4 | Qwen/Qwen2.5-72B-Instruct | 65.75 | 77.22 | 73.78 | 63.83 | 77.77 | 87.55 | 88.51 | 63.49 | 50.06 | 70.74 | 55.90 | 44.19 | 37.20 | 72.75 | 57.51 |\n| #5 | Applied-Innovation-Center/AIC-1 | 65.37 | 73.33 | 72.02 | 77.52 | 76.11 | 88.13 | **90.61** | 56.36 | 53.75 | 68.96 | **62.11** | 50.78 | 28.05 | 69.58 | 47.83 |\n| #6 | Qwen/Qwen3.5-122B-A10B | 64.84 | 74.44 | 73.17 | 37.78 | 81.46 | 86.18 | 86.97 | 64.01 | 47.04 | 55.11 | 50.90 | 52.49 | 65.24 | 72.43 | 60.54 |\n| #7 | Sakalti/Ultiima-72B | 64.49 | 78.33 | 72.28 | 68.79 | 76.75 | 83.70 | 89.08 | 60.44 | 44.58 | 69.12 | 46.91 | 42.25 | 39.02 | 74.07 | 57.56 |\n| #8 | meta-llama/Llama-3.3-70B-Instruct | 63.96 | 77.22 | 71.57 | 78.05 | 77.95 | 88.28 | 85.63 | 67.44 | **56.25** | 64.00 | 51.13 | **54.86** | 27.44 | 71.16 | 24.43 |\n| #9 | Qwen/Qwen2.5-32B-Instruct | 63.26 | 70.56 | 68.76 | 75.80 | 72.07 | 81.03 | 85.82 | 53.78 | 48.08 | 69.27 | 56.94 | 36.51 | 34.15 | 72.75 | **93.10** |\n| #10 | FreedomIntelligence/AceGPT-v2-32B-Chat | 61.14 | 76.67 | 70.62 | 79.79 | 74.46 | 84.88 | 86.97 | 63.89 | 49.96 | 71.46 | 56.04 | 47.32 | 23.78 | 54.50 | 15.56 |\n\n- **Scale does not guarantee best performance.** The top 10 spans models from 32B to 397B parameters, with several mid-size models outperforming larger ones on specific domains.\n- **Arabic-specialized models lead on cultural and linguistic tasks.** Jais-2-70B-Chat ranks highest on ArabicMMLU and ArabCulture, while Karnak leads on 3LM STEM and ArabLegalQA.\n- **Coding remains the hardest domain for Arabic-specialized models.** The top HumanEval+ and MBPP+ scores belong to multilingual models, with Qwen3.5-397B leading both.\n\n### The Size-Performance Relationship\n\nAcross the full leaderboard (46 models), a clear but imperfect size-performance correlation emerges. However, there are interesting exceptions:\n\n[![c64aafc7-1](https://cdn-uploads.huggingface.co/production/uploads/66c8620a79b42e5c941b0265/KPdYTBwzMvJEyALTYfGji.png)](https://cdn-uploads.huggingface.co/production/uploads/66c8620a79b42e5c941b0265/KPdYTBwzMvJEyALTYfGji.png)\n\n- Arabic-specialized models often outperform size-matched multilingual models\n- Instruction-tuned models consistently outperform their base counterparts except for Qwen3\n- Some smaller Arabic-specialized models (Fanar-1-9B, ALLaM-7B) outperform much larger multilingual models on specific domains\n\n* * *\n\n## ๐ŸŒŸ What Makes QIMMA Different\n\nTo summarize the distinctive properties of QIMMA:\n\n| Property | Details |\n| ---------------------------- | ------------------------------------------------------------------------- |\n| **Quality-first philosophy** | Validation runs *before* evaluation, not as an afterthought |\n| **Multi-model validation** | Two LLMs with different training + human review for flagged cases |\n| **99% native Arabic** | Avoids translation artifacts almost entirely |\n| **Multi-domain, multi-task** | 7 domains, 3 task types (MCQ, QA, code), 109 subsets |\n| **Code evaluation** | First Arabic leaderboard to include code generation |\n| **Full transparency** | Per-sample inference outputs publicly released, not just aggregate scores |\n| **LightEval-based** | Unified, reproducible evaluation codebase |\n| **Dialectal awareness** | Explicit handling of MSA vs. dialectal variation in prompts and rubrics |\n\n* * *\n\n## ๐Ÿ”— Resources\n\n- ๐Ÿ† **Leaderboard**: [QIMMA Leaderboard](https://huggingface.co/spaces/qimma/leaderboard)\n- ๐Ÿ’ป **Code**: [GitHub](https://github.com/tiiuae/QIMMA-leaderboard.git)\n- ๐Ÿ“„ **Paper**: [Are Arabic Benchmarks Reliable? QIMMA's Quality-First Approach to LLM Evaluation](https://arxiv.org/pdf/2604.03395)\n\n* * *\n\n## ๐Ÿ”– Citation\n\n```bibtex\n@misc{alqadi2026arabicbenchmarksreliableqimmas,\n title={Are Arabic Benchmarks Reliable? QIMMA's Quality-First Approach to LLM Evaluation}, \n author={Leen AlQadi and Ahmed Alzubaidi and Mohammed Alyafeai and Hamza Alobeidli and Maitha Alhammadi and Shaikha Alsuwaidi and Omar Alkaabi and Basma El Amel Boussaha and Hakim Hacid},\n year={2026},\n eprint={2604.03395},\n archivePrefix={arXiv},\n primaryClass={cs.CL},\n url={https://arxiv.org/abs/2604.03395}, \n}\n```\n", "dateCreated": "2026-10-19T01:26:53.998049939+00:00", "date_created": "2026-04-21T10:09:58+00:00", "date_modified": "2026-04-21T10:09:58+00:00", "date_published": "2026-04-21T10:09:58+00:00", "description": "QIMMA validates benchmarks before evaluating models, ensuring reported scores reflect genuine Arabic language capability in LLMs. If you've been tracking Arabic LLM evaluation, you've probably noticed a growing tension: the number ofโ€ฆ", "external_url": "https://huggingface.co/blog/tiiuae/qimma-arabic-leaderboard", "git": { "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e", "path": "items/huggingface-co/2026/04/2026-04-21-qimma-qimw-mountain-a-quality-first-arabic-llm-leaderboard.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/huggingface-co/2026/04/2026-04-21-qimma-qimw-mountain-a-quality-first-arabic-llm-leaderboard.md" }, "id": "urn:aggr:item:0fb8f0ec016bb8df6e71241baf0b8b5497b92e82", "isBasedOn": { "@id": "https://huggingface.co/blog/tiiuae/qimma-arabic-leaderboard" }, "mainEntity": { "@id": "https://huggingface.co/blog/tiiuae/qimma-arabic-leaderboard", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-04-21-qimma-qimw-mountain-a-quality-first-arabic-llm-leaderboard/#webpage" }, "dateModified": "2026-04-21T10:09:58+00:00", "datePublished": "2026-04-21T10:09:58+00:00", "headline": "QIMMA ู‚ูู…ู‘ุฉ : A Quality-First Arabic LLM Leaderboard", "image": { "@type": "ImageObject", "caption": null, "height": 90, "url": "https://cdn-uploads.huggingface.co/production/uploads/659bc8a7b0f43ed69f0b2300/KdH3z5dRfyIAyxgGPN5Ey.png", "width": 160 }, "timeRequired": "PT8M", "url": "https://huggingface.co/blog/tiiuae/qimma-arabic-leaderboard", "wordCount": 1634 }, "name": "Archived snapshot: QIMMA ู‚ูู…ู‘ุฉ : A Quality-First Arabic LLM Leaderboard", "preview": { "alt": null, "color": null, "height": 90, "placeholder": { "data_url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII=", "hash": "" }, "url": "https://cdn-uploads.huggingface.co/production/uploads/659bc8a7b0f43ed69f0b2300/KdH3z5dRfyIAyxgGPN5Ey.png", "width": 160 }, "reading_minutes": 8, "replicated_at": null, "source": "huggingface-co", "source_name": "Hugging Face - Blog", "summary": null, "tags": [], "temporalCoverage": "2026-10-19T01:26:53.998049939+00:00", "title": "QIMMA ู‚ูู…ู‘ุฉ : A Quality-First Arabic LLM Leaderboard", "url": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-04-21-qimma-qimw-mountain-a-quality-first-arabic-llm-leaderboard/", "word_count": 1634 }