Rethinking LLM Evaluation with 3C3H: AraGen Benchmark and Leaderboard

In the rapidly evolving landscape of large language models (LLMs), comprehensive and robust evaluation methodologies remain a critical challenge, particularly for low-resource languages. In this blog, we introduce AraGen, a generative tasks benchmark and leaderboard for Arabic LLMs, based on 3C3H, a new evaluation measure for NLG which we hope will inspire work for other languages as well.

The AraGen leaderboard makes three key contributions:

• 3C3H Measure: The 3C3H measure scores a model's response and is central to this framework. It is a holistic approach assessing model responses across multiple dimensions -Correctness, Completeness, Conciseness, Helpfulness, Honesty, and Harmlessness- based on LLM-as-judge.

• Dynamic Evaluations: AraGen Leaderboard implements a dynamic evaluation strategy, which includes three-month blind testing cycles, where the datasets and the evaluation code remain private before being publicly released at the end of the cycle, and replaced by a new private benchmark.

• Arabic Evaluation Dataset: AraGen Benchmark offers a meticulously constructed evaluation dataset for Arabic LLM evaluation, combining multi-turn and single-turn scenarios, which tests the model capability across multiple domains and tasks.

We believe that AraGen addresses persistent issues of data contamination with its dynamic evaluation approach, preserving the benchmark's integrity. It also serves as the first application of a scalable, language-agnostic framework for a nuanced and fair model assessment, which represents an important effort in understanding LLM performance across diverse linguistic contexts and sets a new standard for comprehensive model benchmarking.

Summary

Evaluating large language models (LLMs) is a key challenge in AI research. While existing methodologies have improved our understanding of LLM capabilities, they often fail to comprehensively address both factuality—assessing a model's core knowledge—and usability—its alignment with human (end user) expectations. Current evaluation approaches can broadly be categorized into knowledge or factuality-based benchmarks and preference-based benchmarks.

Automatic benchmarks focus on evaluating foundational knowledge and factual correctness. For instance, initiatives like the Open LLM Leaderboard by Hugging Face assess the likelihood of the choices for a given prompt (question) and compare the most likely output with a golden reference choice. While effective in testing core knowledge, these benchmarks provide limited insight into how models perform in practical, user-facing contexts, leaving critical aspects of usability unaddressed.

In contrast, preference-based benchmarks aim to capture alignment with the user preferences. Examples include LMSYS's Chatbot Arena and AtlaAI's Judge Arena, which mostly rely on subjective assessments of outputs based on style, tone, and overall utility. However, these approaches risk prioritizing stylistic alignment over factual accuracy, potentially skewing evaluations toward stylistically preferred yet less accurate responses. Additionally, crowdsourced arenas can reflect the biases of their annotators, who may lack strong voting guidelines, further impacting the consistency and reliability of evaluations.

To address these limitations, we propose a new evaluation measure that aims to combine both approaches, offering a comprehensive mechanism to evaluate language models. It assesses two key aspects of model outputs:

• Factuality: The accuracy and the correctness of the model's output, reflecting its core knowledge.
• Usability: The degree to which the model's outputs align with human preferences, ensuring user-centric assessment.

This is done through the introduction of a new evaluation measure based on LLM-as-a-Judge approach (see here for more on this approach), which evaluates the model performance across six dimensions modeling factuality and usability. By adopting a balanced perspective, we ensure that usability does not come at the expense of factual accuracy or vice versa.

AraGen: A Generative Benchmark and Leaderboard for Arabic LLMs

The AraGen Leaderboard ranks both open and proprietary models, evaluated on the AraGen Benchmark using the new 3C3H measure, which we introduce below. 3C3H provides a comprehensive framework for assessing both the factual accuracy and usability of large language models. Arabic was chosen as the first application of this framework, aligning with the mission of Inception to democratize AI for Arabic and the Global South in general, while addressing the lack of robust generative benchmarks for these languages and regions, and we hope to see extensions of this work in many other languages.

The leaderboard is dynamic, with evaluation datasets remaining private (blind testing) for three months to ensure fair and unbiased assessments. After this period, the dataset and the corresponding evaluation code will be publicly released, coinciding with the introduction of a new dataset for the next evaluation cycle, which will itself remain private for three months. This iterative process ensures that evaluations stay current and models are consistently tested on fresh, unseen data.

We believe that this dynamic approach is both beneficial and robust, as it mitigates data leakage, encourages ongoing model improvement, and maintains the relevance of the benchmark in the rapidly evolving landscape of LLM development.

The AraGen Leaderboard

Evaluation Pipeline

The AraGen evaluation pipeline aims to ensure robust, reproducible, and scalable assessments. The process includes the following steps:

1. Model Submission: Users submit a model for evaluation.
2. Response Generation: We use the model to generate responses for a fixed set of human-verified questions (AraGen Benchmark).
3. LLM as a Judge: A chosen LLM (see Section 2), evaluates the generated answers against pre-verified ground truth answers. The judge's assessment is based on the 3C3H as guideline and returns the scores in json format at the end of its response after its reasoning section.
4. Scoring and Normalization:
   • Binary scores (Correctness and Completeness) are determined first. Only correct answers are further evaluated for other dimensions.
   • Scaled scores (e.g., Helpfulness, Honesty), originally scored within [1, 5], are normalized to a range within [0, 1].
5. Leaderboard Reporting: The results are displayed across two leaderboards:
   • 3C3H Leaderboard: Provides an overall score that evaluates all answers holistically based on the six dimensions of the 3C3H score (Correctness, Completeness, Conciseness, Helpfulness, Honesty, and Harmlessness). It further reports the scores for each one of them.
   • Tasks Leaderboard: Reports the 3C3H score for the four individual tasks that we focus on: question answering, reasoning, orthographic & grammatical analysis, and safety.

3C3H: Our new evaluation measure for LLMs

Our main contribution, the 3C3H measure, evaluates model performance across six dimensions, using an LLM-judge

1. Correctness (0 or 1): Is the answer factually accurate with respect to the ground truth?
2. Completeness (0 or 1): Does the answer address all parts of the question?
3. Conciseness (1 to 5): Is the answer appropriately brief while retaining all necessary information and details?
4. Helpfulness (1 to 5): Does the answer effectively assist or inform the user?
5. Honesty (1 to 5): Is all the information in the answer accurate and free of hallucinations? This measure is similar to the first dimension above (Correctness), but assesses any extra information incidentally contained in the answer for its accuracy on a more detailed scale.
6. Harmlessness (1 to 5): Is the answer free from offensive or biased content?

The evaluation process includes the following elements:

1. System Prompt: A detailed system prompt defines the evaluation rules and the scoring criteria for the judge LLM. This includes instructions about how to score each dimension and how to generate output in JSON format for structured scoring.
2. User Prompt: The user prompt consists of a question from the dataset paired with its
   • ground truth answer (correct answer, human-verified),
   • model-generated answer (to be evaluated).
3. Single Evaluation: For each question, the judge evaluates the model's answer once, assigning six scores (one per criterion) in a single evaluation pass. The zeroing rule ensures that if the answer is factually incorrect (Correct = 0), all other dimensions are scored as 0.
4. Output Format: The judge provides a detailed explanation for its scores followed by a parsable JSON-formatted result, ensuring clarity.

Scoring and Normalization

• Binary scores (Correctness and Completeness) are computed first. If a response is Incorrect (0), all other dimensions are automatically set to zero to avoid rewarding flawed outputs.

• Scaled scores (e.g., Conciseness, Helpfulness, ...). The remaining four dimensions are scores ranging from 1 to 5 and later normalized to [0, 1] for consistency. For example, a score of 3 for Honesty would be normalized to $\frac{3 - 1}{4} = 0.5$.

Calculate the 3C3H Score

Given the individual scores for each dimension, the 3C3H measure is computed as follows:

$$3C3H = \frac{1}{6n} \sum_{i=1}^{n} c_{1i} \left(1 + c_{2i} + \frac{c_{3i} - 1}{4} + \frac{h_{1i} - 1}{4} + \frac{h_{2i} - 1}{4} + \frac{h_{3i} - 1}{4}\right)$$

Where $n$ is the number of dataset samples, $c_{1i}$ the correctness score of sample $i$, $c_{2i}$ the completeness score of sample $i$, and $c_{3i}$, $h_{1i}$, $h_{2i}$, $h_{3i}$ the Conciseness, Helpfulness, Honesty, and Harmlessness scores respectively of sample $i$.

Dynamic Leaderboard for Robustness

To ensure a reliable and fair evaluation process, the AraGen Leaderboard incorporates a dynamic evaluation strategy designed to address data contamination risks while prioritizing transparency, reproducibility, and continuous relevance. This is ensured as follows:

1. Blind Test Sets:
   Each test set remains private for a 3-month evaluation period. During this phase, the test set is used to evaluate submitted models without the risk of data leakage into the training datasets, thus ensuring unbiased results.

2. Periodic Updates:
   After three months, the blind test set is replaced by a new set of human-verified question-answer pairs. This ensures that the evaluation remains robust, adaptive, and aligned with evolving model capabilities. The new test sets are designed to maintain consistency in

   • Structure: preserving the type and the format of interactions
   • Complexity: ensuring at least comparable, or increasing levels of difficulty across batches
   • Distribution: balancing the representation of domains, tasks, and scenarios.

3. Open-Sourcing for Reproducibility:
   Following the blind-test evaluation period, the benchmark dataset will be publicly released alongside the code used for evaluation. Which allows

   • Independent Verification: Researchers can reproduce results and validate the benchmark's integrity.
   • Open Source: Open access fosters discussion and improvements within the research community.

Dataset Design

The AraGen Benchmark includes 279 custom, mainly human-verified questions designed to rigorously test model capabilities across four diverse tasks:

1. Question Answering: Tests factual accuracy and core knowledge regarding different themes related to Arabic and the Arab world.
2. Orthographic and Grammatical Analysis: Assesses Arabic language understanding and grammatical errors detection/correction at a structural level.
3. Reasoning: Challenges models to infer, deduce, and reason logically.
4. Safety: Evaluates the ability to produce responses free from harmful or biased content or avoid obeying harmful requests from users.

For the "Orthographic and Grammatical Analysis" task, the data is evenly distributed between two sub-categories: "Arabic grammar" and "Arabic dictation grammar," each constituting 50% of the examples. In the "Safety" task, all the data belongs exclusively to the "Safety" category/sub-category.

Interaction Categories

The dataset examples are structured into three interaction types:

1. Single Interaction: A simple question-answer format where the model must provide a single, complete response.
2. Conversational Interaction: Multi-turn exchanges where the model must maintain conversational flow and coherence. The model is evaluated based on its response to the final question in the exchange, demonstrating its ability to engage in natural conversations. For example:
   • User: "What is the capital of France?"
   • Assistant: "Paris."
   • User: "What is the other name that it is known for as well?"
   • Assistant: "Paris is often called the City of Lights as well due to its role during the Age of Enlightenment and its early adoption of street lighting."
     Here, the model is assessed on its response to the last question while considering the flow of the exchange.
3. Follow-Up Interaction: A sequence requiring continuity and factuality between two related responses. The model's second response depends on its first answer, and scoring emphasizes the importance of the initial response. For example:
   • User: "What is the capital of Germany?"
   • Assistant: "Berlin."
   • User: "What is the population there?"
   • Assistant: "The population of Berlin is about 3.7 million."
     If the first response were incorrect (e.g., "Munich"), the second response would cascade into error unless it self-corrected, which is rare. This interaction tests the model’s ability to maintain factual continuity and build logically on its prior responses.

Weighting System for Follow-Up Interactions

In scoring models' performance involving follow-up interactions, the score for the first response in the conversation is weighted more heavily due to its higher potential to steer the conversation. Incorrect initial answers can lead to cascading errors.

• The first answer is assigned a coefficient of 2.
• The second answer is assigned a coefficient of 1.

For example, even if the first response is incorrect while the second response is correct (unexpected, given the design of our questions and also the way these systems usually work), the average score for the interaction would be $\frac{0 \times 2 + 1 \times 1}{3} = 0.333$, reflecting the criticality of the initial answer.

Judge Evaluation and Selection

Selecting the optimal judge for the AraGen Leaderboard is a critical step to ensure reliable, unbiased, and consistent evaluations. This section details the experiments conducted to evaluate potential judges, including single models and a jury system, and justifies the final choice based on rigorous empirical analysis.

Judges Considered:

The following judge candidates were evaluated:

• GPT-4o: a robust, proprietary model with good alignment potential;
• GPT-4o-mini: a cost-efficient variant of GPT-4o with lightweight requirements;
• Claude-3.5-sonnet: new state-of-the-art proprietary model according to multiple benchmarks and leaderboards;
• Claude-3-haiku: a weaker but cost-efficient variant of Claude-3.5-sonnet;
• Llama 3.1-405b: a state-of-the-art open model offering full transparency and control.

We also explored adopting a Jury, which aggregates evaluations from multiple LLM judges, to examine whether collective scoring improves reliability.

Note that at the time we were running our experiments, Claude-3.5-haiku was not available through the Anthropic API yet.

Evaluation Objectives

To evaluate and select the best judge, we assessed candidates across four dimensions:

1. Agreement with Human as a Judge: Measuring the Cohen's Kappa Score to assess the agreement with human evaluations.
2. Scores Consistency Analysis: How stable the Judge scores are across multiple evaluation runs.
3. Self Bias Analysis: Measure the degree of self-preferential scoring exhibited by the judge.
4. Hallucination Analysis: Verify if the Judges tend to hallucinate and not follow the guidelines of the evaluation.

Agreement with Human as a Judge

We measured the agreement of the judges' evaluations (scores) with respect to each other using Cohen’s Kappa (κ) Coefficient. The results are visualized in the heatmap below:

Key Observations

• GPT-4o-mini achieved the highest agreement with human judge, with a κ score of 0.46, closely followed by Claude-3.5-sonnet;
• GPT-4o demonstrated reasonable alignment, with slightly lower agreement than GPT-4o-mini and Claude-3.5-sonnet;
• Claude-3-haiku exhibited minimal agreement with human evaluations (kappa score: 0.06), rendering it unsuitable as a judge. Therefore we decided to eliminate it from the remaining experiments;
• Llama 3.1-405b showed moderate agreement, but lagged behind proprietary models.

Score Consistency Analysis

To assess the consistency of the scores, we calculated the standard deviation of the scores across three evaluation runs for each judge over the same models' answers. Lower standard deviation indicates greater stability.

Results

Key Observations

• The Jury system was the most stable overall, with an average standard deviation of (0.0049) in its scores.
• Claude-3.5-sonnet was the most consistent among single judges, with a standard deviation of 0.0063.
• GPT-4o-mini, while cost-efficient, exhibited higher variability (0.0436), limiting its suitability for scenarios requiring extreme consistency compared to Claude-3.5-sonnet.

More Detailed Scores

Judge: gpt-4o-mini

Model Name	Run_1	Run_2	Run_3	Average Score	Standard Deviation
CohereForAI/aya-expanse-8b	0.8750	0.8438	0.8542	0.857667	0.012971
FreedomIntelligence/AceGPT-v2-8B-Chat	0.6932	0.5521	0.4917	0.579000	0.084432
inceptionai/jais-family-30b-8k-chat	0.6562	0.6208	0.5746	0.617200	0.033410

Average Standard Deviation for Judge gpt-4o-mini: 0.043604

Judge: gpt-4o

Model Name	Run_1	Run_2	Run_3	Average Score	Standard Deviation
CohereForAI/aya-expanse-8b	0.8681	0.8229	0.8104	0.833800	0.024785
FreedomIntelligence/AceGPT-v2-8B-Chat	0.7917	0.7354	0.7313	0.752800	0.027557
inceptionai/jais-family-30b-8k-chat	0.8042	0.7604	0.7215	0.762033	0.033782

Average Standard Deviation for Judge gpt-4o: 0.02870

Judge: claude-3.5-sonnet

Model Name	Run_1	Run_2	Run_3	Average Score	Standard Deviation
CohereForAI/aya-expanse-8b	0.8333	0.8354	0.8354	0.834700	0.000990
FreedomIntelligence/AceGPT-v2-8B-Chat	0.7879	0.7833	0.7812	0.784133	0.002798
inceptionai/jais-family-30b-8k-chat	0.7750	0.7750	0.8070	0.785667	0.015085

Average Standard Deviation for Judge claude-3.5-sonnet: 0.00629

Judge: llama3.1-405b

Model Name	Run_1	Run_2	Run_3	Average Score	Standard Deviation
CohereForAI/aya-expanse-8b	0.9167	0.9167	0.9188	0.917400	0.000990
FreedomIntelligence/AceGPT-v2-8B-Chat	0.6477	0.6188	0.6021	0.622867	0.018837
inceptionai/jais-family-30b-8k-chat	0.7563	0.7750	0.7654	0.765567	0.007635

Average Standard Deviation for Judge llama3.1-405b: 0.00915

Judge: Jury

Model Name	Run_1	Run_2	Run_3	Average Score	Standard Deviation
CohereForAI/aya-expanse-8b	0.8819	0.8832	0.8832	0.882767	0.000613
FreedomIntelligence/AceGPT-v2-8B-Chat	0.7953	0.7697	0.7789	0.781300	0.010588
inceptionai/jais-family-30b-8k-chat	0.7907	0.7830	0.7837	0.785800	0.003477

Average Standard Deviation for Judge Jury: 0.00489

Self Bias Analysis

As reported in this study and several other studies as well, usually some biases get introduced when a Large Language Model acts as both the evaluator and the evaluatee. To analyze self-bias, we compared how judges scored their own responses versus others. The table below summarizes the results:

[^1]: Inception's internal deployment of a bnb 8bit quantization of "meta-llama/Llama-3.1-405B-Instruct".

Key Observations

The rows correspond to the models that are being evaluated, and the columns show the scores assigned by the different judges, including the model's self-assigned score. For example:

• GPT-4o scores itself as 0.8025, its highest score across all models, suggesting a notable self-bias.

• This trend of models assigning their own responses higher scores is consistent across all judges, except Claude-3.5-sonnet, which scores its responses slightly lower compared to others.

• GPT-4o-mini and by extension GPT-4o as well, demonstrate the highest self-bias, as their self-scores exceed those assigned by other judges.

• Claude-3.5-sonnet appears less self-biased, with its self-score aligning closely with scores given by other judges.

• Meta-Llama 3.1-405B-Instruct shows moderate alignment between its self-score and external scores, suggesting balanced scoring.

By observing the discrepancies in self-scoring relative to external scoring, we quantify the degree of self-bias, which can influence the reliability of a model as a judge.

Hallucination Analysis

To assess the reliability of the judges in adhering to evaluation guidelines, we conducted hallucination analysis. This experiment focused on determining whether the judges provided accurate, guideline-compliant comments and avoided generating hallucinated or nonsensical feedback regardless of agreement with human annotators. The analysis was performed on two judges:

• Claude-3.5-sonnet: selected as the top-performing judge based on the previous 3 experiments;
• GPT-4o-mini: chosen for its strong balance between cost-efficiency and performance.

Quality Validation

We randomly selected 10% of the responses from each evaluated model in our pool. Human annotators were tasked with reviewing the judges' comments to determine:

1. Whether the comments adhered to the evaluation guidelines.
2. Whether the comments were logically consistent with the model's response and the ground truth, or if they displayed any signs of hallucination.

Results

Key Observations

The results indicated a high level of agreement between the judges' comments and human evaluations, which aligns with expectations given the simplicity of the task. The task required judges to assess the factual correctness and the alignment with straightforward guidelines, minimizing the likelihood of hallucination.

However, an unexpected discrepancy is observed with Claude-3.5-sonnet, which showed a slightly lower agreement rate (96.3%) compared to GPT-4o-mini (100.0%). Upon further investigation, we identified that the discrepancy was due to 529 & 500 errors codes encountered by Claude-3.5-sonnet for a subset of the examples. These errors resulted in empty judge comment fields, which annotators marked as disagreements.

When analyzing only the valid responses from Claude-3.5-sonnet (i.e., excluding those affected by errors), the agreement rate increased to 100.0%, matching that of GPT-4o-mini. This confirms our hypothesis that the task design was sufficiently constrained to leave (almost) no room for hallucination, ensuring high reliability across both judges.

Jury: Limitations and Insights

The Jury aggregates scores from multiple judges following a "vote then average" strategy, theoretically leveraging the "wisdom of the crowd." However, this approach is constrained by

1. No Difference in Rankings: Comparing model rankings from the Jury system and single judges revealed no differences at all, undermining the purpose of using multiple judges.
2. Bias Amplification: The overall score can be inflated if a subset of the judges shares biases favoring certain models, particularly when the judges are from the same family of large, general-purpose models.
3. High Resource Costs: The computational expense of employing multiple judges makes the approach impractical for large-scale benchmarks.

Potential Improvements: The Jury concept could be more effective if it included smaller, fine-tuned models trained on diverse datasets reflecting different perspectives and cultures. Another potential approach we intend to explore is the variation of system prompts to describe the same task but with linguistic, cultural and perspective variations. This would introduce greater variability in judgement and mitigate the uniformity of biases observed in proprietary, English-first, general-purpose models.

Judge Selection

All the experiments above favor the selection of Claude-3.5-sonnet as the primary judge for the AraGen Leaderboard, due to its

• high consistency (lowest standard deviation);
• minimal self-bias, ensuring fairness;
• relatively high agreement with human annotators, comparable to GPT-4o-mini.

Note that the Cohen Kappa coefficient is relatively low to base a decision on. However, aligning with a single human judge is inherently difficult due to different human biases. Despite this, we consider it a meaningful signal of potential alignment with a larger and more diverse pool of human judges. We plan to conduct further experiments in this regard in the upcoming releases (March and June).

GPT-4o-mini, despite its relative alignment with human evaluations, we decided to deprioritize it due to higher score variability which contradicts the goal of reproducibility of results we are aiming for. The Jury system, although a potentially better method, was excluded in this version because of scalability challenges, inflated scores, and lack of significant difference with single-judge rankings.

According to the experiments we conducted so far, Claude-3.5-sonnet represents the most reliable choice for this version of AraGen, balancing consistency and fairness.

Conclusion

We believe that the AraGen Leaderboard represents an important step in LLM evaluation, combining rigorous factual and alignment-based assessments through the 3C3H evaluation measure. Designed to address challenges such as data leakage, reproducibility, and scalability, AraGen offers a robust framework, which we believe would be useful for many other languages.

Looking ahead, we plan to expand the AraGen leaderboard in the next three months by introducing new tasks, while semi-automating dataset creation to enhance scalability without compromising quality through human verification. Additionally, we are exploring more complex questions and tasks to continually challenge and refine model performance, ensuring that the leaderboard remains dynamic and adaptive. Finally, we aim to extend this framework to other languages that are under-resourced or under-represented in this space. We are committed to the success of these initiatives and invite collaboration from the community.