GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP (2023.emnlp-main)
Copied to clipboard
| Challenge: | Our study examines ChatGPT’s performance on Arabic languages and dialectal varieties. |
| Approach: | They conduct a large-scale automated and human evaluation of ChatGPT, encompassing 44 distinct language understanding and generation tasks on over 60 different datasets. |
| Outcome: | The proposed model outperforms smaller models on Arabic dialects compared to GPT-4's Modern Standard Arabic and Dialectal Arabic (DA) |
Similar Papers
A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets (2023.findings-acl)
Copied to clipboard
Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, Jimmy Huang
| Challenge: | Currently, the evaluation of large language models (LLMs) such as ChatGPT in academic datasets is difficult due to the difficulty of evaluating the generative outputs produced by this model against the ground truth. |
| Approach: | They evaluate ChatGPT across 140 tasks and analyze 255K responses it generates in academic datasets. |
| Outcome: | The proposed model performs well on 140 tasks and generates 255K responses in these datasets. |
ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in natural language processing (NLP) have led to significant breakthroughs in the field. |
| Approach: | They evaluate ChatGPT over multiple tasks with diverse languages and large datasets to provide more comprehensive information for multilingual NLP applications. |
| Outcome: | The proposed model can process and generate texts for multiple languages due to its multilingual training data. |
AL-QASIDA: Analyzing LLM Quality and Accuracy Systematically in Dialectal Arabic (2025.findings-acl)
Copied to clipboard
| Challenge: | Dialectal Arabic (DA) varieties are under-served by language technologies, particularly large language models (LLMs). |
| Approach: | They propose a framework that comprehensively assesses LLMs’ DA modeling capabilities across four dimensions: fidelity, understanding, quality, and diglossia. |
| Outcome: | The proposed framework assesses LLMs’ DA modeling capabilities across four dimensions: fidelity, understanding, quality, and diglossia. |
GPTEval: A Survey on Assessments of ChatGPT and GPT-4 (2024.lrec-main)
Copied to clipboard
| Challenge: | emergence of ChatGPT has generated speculation about its potential to disrupt social and economic systems. |
| Approach: | They analyze prior assessments of ChatGPT and GPT-4 to analyze their language and reasoning abilities, scientific knowledge, ethical considerations and existing evaluation methods. |
| Outcome: | The proposed model performs satisfactorily in science knowledge and can answer open questions. |
Arabic Natural Language Processing (2022.emnlp-tutorials)
Copied to clipboard
| Challenge: | This tutorial provides background information for system developers and researchers working with Arabic in its various forms. |
| Approach: | This tutorial provides the necessary background information for working with Arabic in its various forms. |
| Outcome: | This tutorial will explain various Arabic linguistic phenomena and review the state-of-the-art in Arabic processing. |
AceGPT, Localizing Large Language Models in Arabic (2024.naacl-long)
Copied to clipboard
Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Song Dingjie, Zhihong Chen, Mosen Alharthi, Bang An, Juncai He, Ziche Liu, Junying Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ruoyu Sun, Xiang Wan, Haizhou Li, Jinchao Xu
| Challenge: | Significant concerns emerge when addressing cultural sensitivity and local values. |
| Approach: | They propose a localized Large Language Model (LLM) specifically for Arabic, a language imbued with unique cultural characteristics inadequately addressed by current mainstream models. |
| Outcome: | The proposed model sets the state-of-the-art standard for open Arabic LLMs across various benchmarks. |
A Survey of Code-switched Arabic NLP: Progress, Challenges, and Future Directions (2025.coling-main)
Copied to clipboard
| Challenge: | Code-switching (CSW) is a common linguistic phenomenon in multilingual societies . current literature on CSW in the arab world is limited to the Arabic language . |
| Approach: | They present a review of the literature in the field of code-switched Arabic NLP . they propose recommendations for future research . |
| Outcome: | This review provides a broad perspective on the current literature in the field of code-switched Arabic NLP . it also provides recommendations for future research . |
The Arabic Generality Score: Another Dimension of Modeling Arabic Dialectness (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work addresses this issue by modeling dialectness as a continuous variable . however, ALDi reduces complex variation to a single dimension . |
| Approach: | They propose a way to model Arabic dialectness as a continuous variable . they propose etymology-aware edit distance and a regression model to model AGS . |
| Outcome: | The proposed approach outperforms baselines on a multi-dialect benchmark. |
TounsiBench: Benchmarking Large Language Models for Tunisian Arabic (2025.emnlp-main)
Copied to clipboard
| Challenge: | a dataset of Tunisian Arabic instructions and prompts is used to evaluate LLMs' ability to understand and generate responses in Tunisia . we assess the quality, correctness, relevance, and dialectal adherence of LLM responses . |
| Approach: | They propose a benchmark for evaluating the capabilities of large language models in Tunisian Arabic . they use a dataset of Tunisia Arabic instructions and prompts to evaluate their models . |
| Outcome: | The proposed model can judge quality, correctness, relevance, and dialectal adherence . the model can also generate a leaderboard for the Tunisian Arabic language . |
On Using Arabic Language Dialects in Recommendation Systems (2025.findings-naacl)
Copied to clipboard
| Challenge: | Using natural language processing (NLP) to analyze user reviews in recommendation systems is unexplored. |
| Approach: | They propose to integrate Arabic dialects as a signal in recommendation systems by using explicit and implicit approaches. |
| Outcome: | The proposed approach improves recommendation performance and encourages further research in the Arab multicultural world. |