Safety alignment for Arabic Language Models (ALMs) remains underexplored even as these systems are increasingly deployed across the Middle East and North Africa. Most established safety benchmarks and guard models are English-centric, and evaluations that focus only on Modern Standard Arabic (MSA) can miss vulnerabilities that appear under regional linguistic variation.
Arabic introduces additional safety challenges through dialectal usage, indirect expression, and cultural framing. A model that appears aligned in MSA may behave differently when the same harmful intent is reformulated in a regional variety, so aggregate or MSA-only scores can provide an incomplete picture of safety robustness.
To address this gap, we introduce SalamahBench, a paired, cross-variety Arabic safety benchmark containing 8,270 harmful prompts (49,620 paired instances) aligned with the MLCommons safety taxonomy. The prompts are represented in MSA and five regional Arabic varieties namely: Egyptian, Saudi, Syrian, Lebanese, and Moroccan. The benchmark supports category-aware and dialect-aware evaluation of Arabic model safety.
SalamahBench Creation
SalamahBench follows a curation and validation pipeline with four phases: preprocessing, AI judging, human validation, and dialect translation. In the preprocessing phase, heterogeneous Arabic and English safety datasets are normalized, non-MSA sources are translated into MSA using our translation pipeline, and then the prompt labels are mapped to a unified MLCommons hazard schema. This standardization creates a common basis for subsequent safety review.
In the AI phase, GPT-5 and Claude Sonnet 4.5 independently judge each preprocessed prompt using standardized hazard definitions and output formats. Their independent assessments reduce dependence on a single evaluator, and prompts involving sexual harms receive an additional hierarchical re-classification step to distinguish closely related MLCommons categories.
The automated review is followed by a human validation phase. Human annotators adjudicate disagreements between the two AI judges, verify that each prompt is genuinely harmful, and validate its assigned category. In the final dialect translation phase, the finalized MSA prompts are translated into Egyptian, Syrian, Saudi, Lebanese, and Moroccan using the same translation, critique/revision, and refusal-detection pipeline to preserve semantic intent across varieties.
The finalized benchmark contains 8,270 human-verified MSA prompts and 49,620 total prompt instances across MSA plus five regional Arabic varieties. Because every variety is derived from the same underlying MSA prompt, SalamahBench enables controlled, paired comparisons of safety behavior across dialects and harm domains.
How we evaluate model safety
The main paired evaluation covers three Arabic language models: Fanar 2 (27B), ALLaM 2 (7B), and Karnak 1 (41B), with Qwen 3.8 27B as a multilingual reference; Fanar 1 (9B), Jais 2 (8B), and Falcon H1R (7B) are additionally evaluated on the MSA prompts only. Each SalamahBench prompt is sent to a target model, and the generated Arabic response is then evaluated directly by Qwen3Guard, Llama Guard 4, and PolyGuard. The paper reports individual safeguard results as well as a three-guard majority-vote setting.
Safety performance is measured primarily with Attack Success Rate (ASR), where lower values indicate stronger safety alignment. For Qwen3Guard, the strict ASR counts both Unsafe and Controversial responses, while the loose ASR counts only Unsafe responses. The study also reports macro-averaged ASR across harm categories, Refusal Rate, Dialect Shift (DS) and Category-Specific Dialect Deviation (CSDD).
The safeguards are validated against human gold labels on 1,440 model responses. Qwen3Guard achieves the highest response-safety classification accuracy (84.7%, against 65.3% for Llama Guard 4, 59.0% for PolyGuard, and 67.8% for the majority vote), so the paper uses Qwen3Guard under the strict setting for its main dialect- and category-level analyses.
Results and Key Observations
Across all 49,620 prompt instances, ALLaM 2 has the lowest ASR of the three main models under the three-safeguard majority vote at 1.3%, followed by Fanar 2 at 2.2% and Karnak 1 at 6.9%. Individual safeguards do not always produce the same ranking: PolyGuard, for example, assigns Karnak 1 the lowest ASR at 0.8%, but it detects only 6.7% of unsafe responses in the human gold set, so its ranking should be read with caution. This evaluator sensitivity motivates the paper's safeguard-validation step rather than relying on a single unverified judge.
Dialect-level analysis changes the picture further. Under the Qwen3Guard strict setting, ALLaM 2's macro-ASR moves from 7.4% in MSA to a 6.6% five-dialect average (mean DS −0.8 percentage points), with lower point estimates in four of five dialects, although this mean shift is not statistically distinguishable from zero. Fanar 2 rises from 11.4% to 12.9% (+1.5 pp), while Karnak 1 rises from 21.5% to 25.4% (+3.9 pp); both shifts are significant after correction, showing that MSA results do not reliably predict regional behavior.
Selected CSDD Highlights Across the Three Models
Category-level analysis shows why aggregate scores alone can be misleading. ALLaM 2's Intellectual Property ASR increases in every dialect, from 22.9% in MSA to 28.3%-40.4% across the five regional varieties, even though its overall dialect shift is not distinguishable from zero. Fanar 2's Intellectual Property ASR also rises (30.1% in MSA versus 30.7%-42.2% across dialects), although only the Saudi increase is significant after correction. Karnak 1 remains more exposed overall. These findings support joint analysis of absolute category ASR, DS, and CSDD instead of a single headline score.