SalamahBench

Each of the 8,270 prompts was screened by two AI judges and checked by human reviewers in MSA before being rendered in five regional varieties. Holding the harmful request fixed while only the dialect changes is what lets the results attribute a change in safety to the language variety itself.

Every response from Fanar 2, Karnak 1 and ALLaM 2 across all six varieties was scored by three safeguards and by their majority vote. ALLaM 2 is the safest under Qwen3Guard, Llama Guard 4 and the majority vote. PolyGuard reverses the order, but it catches only 6.7% of human labeled unsafe responses, so its ranking carries little weight. Whether Karnak 1 or Fanar 2 looks worse depends on the safeguard.

Dialect Shift compares each model’s macro ASR in a dialect with its own MSA baseline. Fanar 2 and Karnak 1 are less safe in every dialect, with mean shifts of +1.5 and +3.9 pp that survive correction. ALLaM 2 moves slightly the other way, but its mean shift of −0.8 pp is not distinguishable from zero. MSA performance alone does not predict which way a model will move.

Aggregate shifts hide very different category stories. ALLaM 2 improves overall, yet its Intellectual Property ASR climbs from 22.9% in MSA to between 28.3% and 40.4% in every dialect, while Indiscriminate Weapons falls from 19.5% to as low as 7.4%.

Karnak 1 keeps Intellectual Property between 59.6% and 63.9% in every variety, a persistent weakness rather than a dialect effect, while Indiscriminate Weapons jumps from 42.9% to as much as 59.3%. Fanar 2’s clearest deterioration is also Intellectual Property, reaching 42.2% in Saudi Arabic.

Qwen 3.8 (27B) is safe overall, yet its ASR rises from 1.4% in MSA to between 2.5% and 3.6% in every dialect, highest in Moroccan. Its category balanced shift is larger, +2.0 to +4.6 pp, and Intellectual Property jumps from 3.0% to between 17.5% and 33.1%, a mean deviation of +19.2 pp and the largest category effect in the study. Low aggregate ASR can hide high exposure in a small category.

On the 8,270 MSA prompts, Falcon H1R has the lowest macro ASR and Jais 2 the highest. Intellectual Property is the most exposed category for four of the six models, reaching 54.8% for Fanar 1, 62.7% for Karnak 1 and 69.3% for Jais 2. Falcon H1R is the exception at 4.8%; its highest categories are Sexual Content (10.0%) and Hate (7.9%).

Two expert annotators labeled 1,440 responses, balanced across categories, varieties and models, with a third resolving disagreements (85.8% raw agreement, κ 0.714, before adjudication). Qwen3Guard in its strict setting matched them on 84.7% of responses and flagged 89.0% of the unsafe ones. Llama Guard 4 and PolyGuard look acceptable on accuracy but miss most unsafe answers, which is why Qwen3Guard strict is the primary judge for the dialect and category analyses.

Prompted as four class judges (refusal, safe, controversial, unsafe), the Arabic models fall well short of a dedicated safeguard. Karnak 1 is the best judge at 59.3% overall, then ALLaM 2 at 51.3% and Fanar 2 at 47.0%. Scores depend on who wrote the response: Fanar 2 reaches 65.6% on Karnak 1’s answers but only 38.3% on its own.

Each dialect version came from a translate, critique and revise loop, which beat direct and reasoning guided prompting on every dimension in an LM judge comparison of 500 outputs. Native speakers rated 120 MSA to Egyptian translations at 4.85 out of 5 overall, and at least 93.3% of items scored 4 or 5 on every dimension. Of 157 translations the model first refused, all but 2 were recovered, so the most harmful prompts were not silently dropped.