Researchers introduced Mizan, a benchmark designed to test large language models on Iraqi Arabic and on civic topics relevant to Iraq. Existing Arabic evaluation efforts have largely centered on Modern Standard Arabic, leaving regional dialects and country-specific knowledge thinly covered. The work aims to give a national-level measure of model performance beyond aggregated MSA leaderboards.
A new arXiv paper addresses the mismatch between the Uthmani orthography used in printed Qurans and the Standard (Imla'i) Arabic that most Arabic NLP tools support, since the two forms differ at the byte level. The authors present a corpus-aligned mapping between Uthmani and Standard word forms, along with a deterministic validator for recitation. The work targets improved text processing for Quranic Arabic within existing Arabic language tooling.
A new arXiv paper introduces E-CONAN, a set of benchmarks covering entailment, contradiction and neutral relations for Arabic natural language inference. The authors frame the work as a response to the limited resources available for Arabic compared with English and other well-served languages, noting that inference models are a component of many downstream NLP applications. The datasets are intended to support training and evaluation of Arabic inference systems.
Researchers have released YallaMorph, a benchmark for measuring how well large language models generate morphologically accurate Arabic. It addresses a gap in current Arabic evaluation, which focuses on downstream tasks rather than directly testing whether models can control grammatical forms like inflection and derivation. The work highlights that producing fluent Arabic text does not guarantee correct morphosyntactic output.