Arabic-first evaluation for production AI systems.

The problem: evaluated in English, deployed in Arabic

The Gulf now has a serious stable of Arabic-capable models: Saudi Arabia's ALLaM (SDAIA, served through HUMAIN and Amazon Bedrock), the UAE's Jais family and its successor Jais 2 (G42's Inception with MBZUAI) and the Falcon line including Falcon-Arabic (TII), and Qatar's Fanar (Ministry of Communications and Information Technology with the Qatar Computing Research Institute) ¹⁴¹⁰. Procurement conversations, however, still lean on global English benchmarks, and even Arabic-aware buyers usually stop at a leaderboard average. That average hides the two gaps that matter in production: the gap between English and Arabic performance, and the gap between Modern Standard Arabic and the dialects in which citizens, customers and front-line staff actually write.

Both gaps are now measurable with public instruments. The AraDiCE benchmark suite (Qatar Computing Research Institute, 2024) produced carefully post-edited Egyptian and Levantine versions of standard benchmarks precisely so that the dialect effect could be isolated ⁵. The Fanar technical report, which evaluates several regional models on AraDiCE, shows the pattern cleanly: on the PIQA commonsense task, moving the same questions from MSA to Levantine costs Fanar-1-9B-Instruct 8.7 points and ALLaM-7B-Instruct-preview 6.6 points ⁴. Exhibit 2 plots the numbers. The direction is consistent across models; the size of the drop is model-specific — which is exactly why it must be measured per system, not assumed.

What the public benchmark landscape actually covers

Exhibit 1 maps the principal public instruments. Three observations follow from it. First, the centre of gravity is knowledge measured by multiple choice: ArabicMMLU's 14,575 questions across 40 tasks ³, the AlGhafa suite's classification and reading-comprehension tasks ⁹, school-exam sets such as EXAMS, and human-translated MMLU variants. Second, the aggregation layer exists and is credible: the Open Arabic LLM Leaderboard, launched in May 2024 by 2A2I, TII and Hugging Face and re-based in 2025 as OALL v2, runs a fixed suite on a common harness and publishes everything ². Third, the coverage thins precisely where production risk concentrates: AraTrust is the suite's one safety-and-truthfulness instrument ²; dialect coverage rests largely on AraDiCE-style adaptations ⁵; and no public benchmark covers an institution's own document formats, terminology, workflows or refusal policy — by construction, since public benchmarks must be general.

The governance anchors: why this is compliance-adjacent, not a quality preference

In the UAE's banking sector the model-validation obligation is explicit. The Central Bank of the UAE issued its final Model Management Standards and Guidance on 21 December 2022, applying to all banks and branches operating in the UAE; the Standards cover model governance, data management, development, implementation, usage and independent validation across the model lifecycle, and required banks to submit a gap assessment and remediation plan within six months ⁶. The CBUAE has since issued guidance on consumer protection and the responsible adoption of AI and machine learning by licensed financial institutions, which expects AI governance to align with the Model Management Standards, meaningful human oversight of consequential decisions, continuous monitoring and validation, and equivalent standards for third-party AI providers ⁷. A bank deploying an Arabic-language agent therefore already owes its validator evidence of fitness in Arabic; an English benchmark report does not discharge that duty.

In Saudi Arabia the anchor is SDAIA. The authority published its Generative AI Guidelines for government entities on 10 January 2024 and a companion edition for the public on 11 January 2024, covering responsible use, risk identification and mitigation, and alignment with SDAIA's AI Ethics Principles and the Personal Data Protection Law ⁸. The guidelines are guidance rather than statute, but they define what 'responsible adoption' means for the Kingdom's government buyers — and government buyers write those expectations into procurement. Exhibit 3 summarises the instruments. The practical reading for vendors: evaluation evidence in the language of deployment is becoming part of the compliance file, whether the file is addressed to a central-bank validator, a data office or a ministry's procurement committee.

Seven design principles for an Arabic-first harness

The harness that satisfies both the engineering and the governance requirement can be specified in seven principles (the authors' synthesis). None requires inventing new science; all require discipline.

• Dialect-by-task coverage, not a single Arabic score. Evaluate each production task in MSA and in each dialect the user base actually writes, following the AraDiCE method of post-edited parallel sets, and report the matrix rather than the average ⁵.
• Task-grounded private test sets. Draw evaluation items from the institution's own documents, tickets and transcripts (with data-protection clearance), because public benchmarks cannot represent the institution's formats or terminology — and keep them private so they cannot leak into training data ¹¹.
• Contamination control. Treat any public benchmark score as an upper bound: public sets circulate in training corpora, so complement them with freshly authored or held-out items, and date-stamp every set ²¹¹.
• Safety and refusal testing in Arabic. Jailbreaks, harmful-content probes and refusal-policy tests must be authored in Arabic, including dialect and transliteration variants; AraTrust is the public starting point, not the finish line ².
• Human adjudication by Arabic speakers. Generative outputs need rubric-based human scoring by evaluators who read the users' Arabic, with inter-rater agreement tracked; automatic metrics alone are difficult to defend in front of a validator ⁴¹¹.
• Versioned, reproducible scoring. Pin the harness version, model version, prompts and decoding parameters for every run, the way the open leaderboards pin theirs, so that results can be reproduced at audit ².
• Publishable methodology, private data. The method should survive publication even where the test items cannot — which is precisely the standard a model-risk function applies to any validation ⁶.

Implications

For buyers. Require the dialect-by-task matrix in every AI procurement where Arabic users are in scope, and require the vendor to state which public benchmarks it reports, which versions, and what it holds private. Treat an English-only evaluation annex as a gap, not a formality. For banks, connect the AI evaluation file to the existing model-validation function rather than building a parallel one — the CBUAE Standards already define the governance container ⁶⁷.

For vendors. Build the harness once, as infrastructure, and amortise it across engagements: the public layer (OALL suite, ArabicMMLU, AraDiCE-style dialect sets) is reusable by construction, and the private layer is a template instantiated per client ²³⁵. The firms that can hand a validator a signed Arabic evaluation dossier will find that it functions as a commercial differentiator precisely because it is scarce.