Four Arabic AI Models on Three UAE Tasks: Where the Benchmarks Stop

Jais, Falcon-H1 Arabic, K2 Think and Copilot, run against three Abu Dhabi government workflows and against the Arabic benchmarks that have to stand in for them

AsiaPresswire – The task is a trade licence renewal. A Dubai distributor wants to add two activities to an existing licence, and the application arrives as a scanned Arabic form, a PDF attachment listing the new activities in mixed Arabic and English, and an email thread in which the applicant switches language twice per paragraph. A ministry officer opens it, and the question is whether an AI agent can prepare the response.

This is the work that the UAE has committed to automating at a scale no other government has attempted. The federal programme aims to convert 50 percent of government operations, procedures and services to agentic AI models within two years. Abu Dhabi’s Digital Strategy 2025–2027 carries AED 13 billion and targets more than 200 AI solutions, with 100 already deployed across 40-plus government entities by September 2025 and 95 percent of 30,000 employees trained. Microsoft 365 Copilot sits on 35,000 desks across 27 entities. The pipeline includes more than 1,000 agents handling document processing, public queries and policy analysis.

So the model choice is not a research question in the UAE. It is a procurement question, asked several thousand times, and the answer has to survive an audit. Four candidates are in play, and the published evidence on them is uneven in a way that matters.

What the four candidates actually are

They are not the same kind of object, which is the first thing the comparison tables on vendor sites tend to blur.

Candidate Builder Size Licence Where it can run
Jais G42, Inception, MBZUAI 70B in the current generation, trained on ~1.6T tokens Partially open G42 Cloud, Khazna, OCI dedicated region
Falcon-H1 Arabic Technology Innovation Institute, Abu Dhabi 3B, 7B and 34B, hybrid Mamba-Transformer Open weights Self-hosted, or any of the above
K2 Think MBZUAI Institute of Foundation Models, G42, Cerebras 70B reasoning model on the K2-V2 base Open Same in-country options
Microsoft 365 Copilot Microsoft, with Inception42 Catalyst underneath Not disclosed Commercial SaaS Azure UAE with Advanced Data Residency

 

Two more belong in the frame even though they are not UAE-built: Fanar, the 27B multimodal model from Qatar’s QCRI, and ALLaM, the Saudi model now folded into the state-backed Humain effort. A ministry that insists on Gulf sovereignty has four options, not two, and a ministry that insists on Arabic quality has to look at all six.

The ninth-month addition changes the calculus again. In September 2026 MBZUAI’s Institute of Foundation Models released K2 Horizon, six models from 0.9 billion to 375 billion parameters, all fully open including weights, code, training data and method. That is the largest fully-open release from the region, and it means a government department can now fine-tune a 375B-class model on its own procurement archive without asking anyone’s permission.

Task one: the reply in Emirati

Start with the easiest-sounding task and the one that fails most often in practice: writing a reply to that distributor’s email thread, in Arabic, matching the register the applicant used.

The published yardstick here is ArabicMMLU, a multiple-choice knowledge benchmark built from Arabic school and university exams. It measures Modern Standard Arabic comprehension. It does not measure whether a model can produce a reply that an Emirati official would sign.

Model ArabicMMLU AlGhafa EXAMS Mean across six suites
Arabic.AI LLM-X 76.2% 88.5% 82.3% 80.4%
Qwen3 235B 74.1% 85.2% 80.1% 78.6%
GPT-4 baseline 72.5% 85.1% 79.8% 78.3%
Falcon-Arabic 68.9% 79.1% 75.4% 72.9%
ALLaM 70B 64.7% 73.2% 70.1% 68.3%
Jais-chat 30B 62.3% 71.4% 68.2% 66.4%

 

Read the last row carefully, because it is the trap in almost every comparison published this year. Jais-chat 30B is a previous generation. The current Jais runs to 70 billion parameters on roughly 1.6 trillion tokens, and its ArabicMMLU score is not in this table because the benchmark run on it has not been published in the same form. Anyone ranking Jais below ALLaM on the strength of that 62.3 percent is comparing a 2024 model against a 2025 one and calling it a family verdict.

The other public yardstick is the Open Arabic LLM Leaderboard, where Falcon-H1 Arabic leads. The 7B version averages 71.47 percent across the leaderboard’s tasks; the 34B version reaches 75.36 percent and beats several models above 70 billion parameters. That result is the strongest single Arabic data point produced in the Gulf this year, and it comes from the smallest of the four UAE candidates.

What none of these scores tells you is dialect. The benchmark suite that does is AraDice, which tests Gulf, Levantine and North African comprehension separately, alongside 3LM for science and maths reasoning and ArabCulture for cultural context. AraDice results are not published for most of these models. Until they are, the honest position on dialect is that Jais was built for Arabic-English code-switching and is the only one of the four whose design brief names that problem, and the rest are assumed to handle it because they score well on MSA.

Task two: the form that arrives as a PDF

The second task is the actual bottleneck in that ministry queue, and no leaderboard tests it.

A scanned Arabic form with a handwritten amendment, a stamped attachment, and a table of activity codes that must be mapped against a controlled vocabulary is not a language task. It is a document pipeline: optical character recognition on Arabic script with diacritics and ligatures, layout analysis to recover the table, entity extraction to pull the applicant’s details, and a validation step against the ministry’s registry. The language model sits at the end of that chain, not the start.

Pipeline stage What decides success Which benchmark covers it
Arabic OCR with handwriting Character error rate on handwritten script None of the four
Table recovery from a scan Layout model, not the LLM None
Activity-code mapping Retrieval against ministry taxonomy None
Draft reply in Arabic Register, dialect, code-switching ArabicMMLU partially, AraDice fully
Validation before submission Rules engine, human sign-off Not a model question

 

This is where the deployment numbers start to matter more than the benchmark numbers. Copilot at 35,000 seats is not winning because its underlying model tops a table. It is winning because it arrives attached to Word, Outlook and Teams, which is where the documents already are, and because it runs with Advanced Data Residency so that processing stays inside the UAE. A sovereign model that requires a separate application, a separate login and a separate document store has to beat that convenience by a wide margin, and no published evaluation shows that it does.

The counter-argument is the one every chief information officer in the Gulf now makes: the convenience is the lock-in. A ministry whose procurement archive, correspondence and case files sit inside a foreign vendor’s tenancy has outsourced a function it cannot audit. That is why Inception42’s Catalyst platform exists — to let agencies build agents that can run inside Copilot while the agent logic and data stay sovereign. The chief executive, Ashish Koshy, described the design to AGBI in July: agents built in Catalyst can be used in Copilot and agents built in Copilot can be used in Catalyst, so the department does not build the same tool twice.

Task three: the forty-page tender

The third task is the one the 50 percent target actually depends on, and the one where the four candidates diverge most.

A procurement officer receives a forty-page Arabic tender, must extract the compliance clauses, cross-check them against standing financial instructions, and draft an evaluation memo. That is not a question-and-answer task. It is a sequence: read, retrieve, compare, draft, check. It is what “agentic” means in the federal programme, and it is where the architecture of each candidate becomes visible.

K2 Think is the only one of the four built specifically for this shape of work. It is a 70B reasoning model, released with Cerebras and MBZUAI’s Institute of Foundation Models, designed to do multi-step reasoning at a fraction of frontier-model compute cost. For a ministry running thousands of these a day, cost per case is the number that decides whether the programme scales or stalls at pilot.

Falcon-H1 Arabic’s hybrid Mamba-Transformer architecture matters here for a different reason: long documents. Attention cost grows with context length, and Mamba layers do not. A 40-page tender with annexes is a long-context problem before it is a reasoning problem, and the 34B model’s leaderboard lead was achieved at a size that can be served on modest hardware.

Jais is the strongest fit for the drafting step, because the memo will be written in the register the ministry actually uses, which is Arabic with English technical terms embedded mid-sentence. Code-switching is the specific capability Jais was trained for. It is the weakest evidence base for the retrieval step, because no published evaluation tests Arabic retrieval against a government taxonomy.

Copilot, with Catalyst behind it, is the only one of the four with a documented oversight layer. Koshy’s description of the command centre — a human supervises the agents assigned to them, whether built in Copilot or Catalyst, and can see how often those agents fail — is the answer to the question every auditor asks first. It is also the only one where the failure rate is visible at all.

The trade-off that is not in the brochure

Put the four side by side on the dimensions that decide a government purchase rather than a model purchase, and the pattern is not a ranking. It is a set of trades.

Dimension Open sovereign models Commercial agent layer
Arabic quality on published benchmarks Documented, comparable, improving Not disclosed
Dialect coverage Partial, one vendor names it Assumed, untested publicly
Document pipeline Buyer must assemble it Included
Data residency In-country by construction Contractual, Azure UAE
Auditability of failure Depends on the buyer’s own harness Built-in command centre
Cost per case at scale Compute only Per-seat, per-month
Exit cost Low High

 

The UAE has no single AI act. Regulation is distributed across data protection law, sector rules and emirate-level guidance, which means a ministry cannot point to a statute that tells it which model to buy. It defaults to the procurement criteria it can defend: where the data sits, who can be audited, and what happens when the model is wrong.

That default explains a deployment pattern that looks inconsistent from outside. The same government that runs 35,000 Copilot seats also funded a 375-billion-parameter open release and a 1 GW AI campus. It is not hedging. It is buying the agent layer from a vendor whose failure modes are documented, while building the sovereign capacity to replace that layer before the contract renews.

Koshy’s own framing of the risk is unusually direct for a vendor: “If the expectation is that while we are experimenting with AI, we will not have mistakes made, I think that does not exist. It’s how quick we can learn from those mistakes and scale that in a manner that we become transparent.” That is the standard the programme will be judged against, and it is not a benchmark score.

Two adoption numbers, fourteen points apart

Before any of this gets bought, someone has to answer a prior question: how much of the workforce already uses AI? The UAE has two published answers for 2026 and they do not agree.

Microsoft’s Global AI Diffusion Report puts UAE adoption at 73.3 percent in the second quarter of 2026. A separate figure carried by The National and picked up in Chinese business coverage puts it at 59.4 percent. Same country, same year, fourteen points apart — and both circulate in the same procurement decks.

Figure Source Likely base What it implies for a buyer
73.3% Microsoft Global AI Diffusion Report, Q2 2026 Users of Microsoft-scale consumer and work accounts Adoption is solved; buy orchestration
59.4% The National, citing survey data, 2026 Survey respondents, method undisclosed A third of staff need training first

 

The gap is almost certainly a denominator difference, not a disagreement about reality. Diffusion studies built from product telemetry count accounts that touched a generative feature at least once in a window; population surveys count people who say they use AI. Those are different populations and different thresholds, and a single sign-in can move the first by a percentage point.

Why it matters operationally: a ministry that believes 73 percent will buy agent orchestration and skip the training budget. A ministry that believes 59 percent will buy training and delay the agents. Both will spend a year on the wrong half of the problem, because neither number measures the thing that predicts success — whether the people who must sign off on an AI-drafted decision trust it enough to sign.

The variable that actually predicts deployment success is not adoption. It is the number of staff who have used the tool on a live case with a supervisor watching. Abu Dhabi’s own disclosure is more useful than either headline: 95 percent of 30,000 employees trained, 35,000 Copilot seats live, 100 use cases in production. Train, seat, case. Only the third one ships.

Data basis: what I ran, and what I did not

This is the part most tool comparisons omit, so it goes here rather than in a footnote.

I did not run these four models on the three tasks above. I have no access to a UAE ministry’s licence registry, no Arabic handwriting test set, and no licensed copy of the ministry’s procurement taxonomy. Any table of “actual outputs” I produced would be a demonstration, not an evaluation, and a demonstration on tasks I invented would tell a reader nothing about the tasks that matter.

What I did instead was assemble every published number that bears on the three tasks: the ArabicMMLU and six-suite results, the Open Arabic LLM Leaderboard averages for Falcon-H1 Arabic, the parameter counts and training volumes from the model cards, the deployment counts from Abu Dhabi’s Department of Government Enablement, and the architectural claims from each builder. Where a number does not exist, the tables say so rather than substituting a plausible one.

That method has a clear limit. It can tell you which model has published evidence of Arabic capability and which does not. It cannot tell you which one will handle an Emirati applicant’s handwriting on a scanned form, because nobody has published that test. A buyer who needs the second answer has to run it, and should budget for that run rather than assume the leaderboard settles it.

What I would deploy, and in what order

If the constraint is a real ministry queue and a two-year clock, the sequence that survives contact with procurement looks like this.

  • Month one to three: the document pipeline, not the model. Fix Arabic OCR, table recovery and the taxonomy lookup. Every one of the four candidates performs better than a broken pipeline and worse than a working one, and the pipeline is where the failure rate actually lives.
  • Month three to six: Copilot for the drafting step, on the 35,000-seat tenancy already paid for. The marginal cost is zero and the register is close enough for a first draft that a human edits.
  • Month six to twelve: Falcon-H1 Arabic 34B self-hosted for the long-document step. It has the published Arabic evidence, the open weights, the long-context architecture, and a size a ministry can serve without a GPU allocation fight.
  • Month twelve onward: Jais for citizen-facing Arabic correspondence, where code-switching is the actual failure mode and the only model designed for it. Test AraDice first; if the dialect scores are not published by then, run them internally.
  • Never: a single-vendor agent layer with no exit. The 1,000-agent pipeline is a five-year commitment to whoever supplies it. The exit cost is the number to negotiate, not the seat price.

The 50 percent target will be met or missed on the first item, not the fourth. Arabic OCR on a scanned government form is unglamorous, unbenchmarked, and the reason a ministry queue does or does not move. No model release fixes it, and no leaderboard will ever rank it.

There is a second thing the ranking hides. Every candidate in this comparison is good enough to draft a reply that a competent officer can fix in four minutes. None of them is good enough to draft a reply that a competent officer can sign without reading. The distance between those two standards is the entire adoption problem, and it is measured in minutes saved per case, not in percentage points on ArabicMMLU. A ministry that instruments that number will know within a quarter whether the 50 percent target is reachable. One that tracks leaderboard position will not know until the two years are up.

Sources