Researchers introduced Mizan, a benchmark designed to test large language models on Iraqi Arabic and on civic topics relevant to Iraq. Existing Arabic evaluation efforts have largely centered on Modern Standard Arabic, leaving regional dialects and country-specific knowledge thinly covered. The work aims to give a national-level measure of model performance beyond aggregated MSA leaderboards.
Researchers detail Rosetta, their entry for Subtask 1 of the AlexandriaX shared task, which covers context-aware translation of English dialogue into dialectal Arabic, competing in both the constrained and unconstrained tracks. The system applies a LoRA adapter fine-tuned on top of NileChat to handle dialect variation in conversational translation.
Researchers have released YallaMorph, a benchmark for measuring how well large language models generate morphologically accurate Arabic. It addresses a gap in current Arabic evaluation, which focuses on downstream tasks rather than directly testing whether models can control grammatical forms like inflection and derivation. The work highlights that producing fluent Arabic text does not guarantee correct morphosyntactic output.
A new arXiv paper introduces E-CONAN, a set of benchmarks covering entailment, contradiction and neutral relations for Arabic natural language inference. The authors frame the work as a response to the limited resources available for Arabic compared with English and other well-served languages, noting that inference models are a component of many downstream NLP applications. The datasets are intended to support training and evaluation of Arabic inference systems.