Generalists vs. Specialists: Evaluating Large Language Models for Urdu (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Urdu is underrepresented in natural language processing, yet it is underserved. |
| Approach: | They compare general-purpose models with special-purpose ones that have been fine-tuned on specific tasks. |
| Outcome: | The proposed models outperform general-purpose models on seven classification and seven generation tasks. |
Similar Papers
UrBLiMP: A Benchmark for Evaluating the Linguistic Competence of Large Language Models in Urdu (2026.findings-acl)
Copied to clipboard
| Challenge: | Evaluating how large language models capture grammatical structure of low-resource languages remains underexplored. |
| Approach: | They evaluate a set of 5,696 minimal pairs that contrast grammatical acceptability across ten core syntactic and morpho-syntactical phenomena in Urdu. |
| Outcome: | The proposed framework compares multilingual models with the proprietary model . the proposed framework achieves the highest average accuracy on regular phenomena . |
Benchmarking Large Language Models for Persian: A Preliminary Study Focusing on ChatGPT (2024.lrec-main)
Copied to clipboard
Amirhossein Abaskohi, Sara Baruni, Mostafa Masoudi, Nesa Abbasi, Mohammad Hadi Babalou, Ali Edalat, Sepehr Kamahi, Samin Mahdizadeh Sani, Nikoo Naghavian, Danial Namazifard, Pouya Sadeghi, Yadollah Yaghoobzadeh
| Challenge: | a new study examines the efficacy of large language models (LLMs) for Persian . ChatGPT and LLMs have shown remarkable performance in English, but their efficiency for low-resource languages remains an open question. |
| Approach: | They present a benchmarking study of large language models (LLMs) for Persian . they focus on GPT-3.5-turbo, but also GPT-4 and OpenChat-3.5 . |
| Outcome: | The proposed model performs better in Persian than other low-resource languages . the study is the first comprehensive benchmarking of large language models . |
HLU: Human Vs LLM Generated Text Detection Dataset for Urdu at Multiple Granularities (2025.coling-main)
Copied to clipboard
| Challenge: | Using large language models (LLMs) to generate human-like text has raised concerns about misuse, especially in low-resource languages like Urdu. |
| Approach: | They propose a dataset that contains documents, paragraphs, and sentences . they conducted human evaluations and automated evaluations . |
| Outcome: | The proposed dataset shows that distinguishing between human and machine-generated text is challenging for both humans and LLMs. |
Investigating Large Language Models for Complex Word Identification in Multilingual and Multidomain Setups (2024.emnlp-main)
Copied to clipboard
Răzvan-Alexandru Smădu, David-Gabriel Ion, Dumitru-Clementin Cercel, Florin Pop, Mihaela-Claudia Cercel
| Challenge: | Large language models (LLMs) are popular in the Natural Language Processing community because of their versatility and capability to solve unseen tasks in zero/few-shot settings. |
| Approach: | They investigate the use of large language models in CWI, LCP, and MWE settings by evaluating their use in zero-shot, few-shot and fine-tuning settings. |
| Outcome: | The proposed models struggle in certain conditions or achieve comparable results against existing methods. |
A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations (2024.emnlp-main)
Copied to clipboard
Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty, Jimmy Huang
| Challenge: | Large Language Models (LLMs) have gained significant attention due to their capabilities in performing diverse tasks across domains. |
| Approach: | They review the primary challenges and limitations causing inconsistencies in evaluations . early models could generate coherent text but limited to simple tasks . |
| Outcome: | The proposed evaluations are reproducible, reliable, and robust. |
MEGA: Multilingual Evaluation of Generative AI (2023.emnlp-main)
Copied to clipboard
Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, Sunayana Sitaram
| Challenge: | Large Large Models (LLMs) have shown impressive performance on many natural language processing tasks such as language understanding, reasoning, and language generation. |
| Approach: | They present a framework for evaluating generative LLMs in the multilingual setting and provide directions for future progress in the field. |
| Outcome: | The proposed framework evaluates generative models on 16 NLP datasets across 70 typologically diverse languages and compares them to state-of-the-art non-autoregressive models. |
AraReasoner: Evaluating Reasoning-Based LLMs for Arabic NLP (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models have shown remarkable progress in reasoning abilities and general natural language processing tasks, yet their performance on Arabic data remains underexplored. |
| Approach: | They compare reasoning-focused LLMs with deepSeek models across 15 Arabic NLP tasks . they use zero-shot, few-shot and fine-tuning to evaluate their capacity for linguistic reasoning . |
| Outcome: | The proposed models outperform strong models on Arabic datasets and are compared with other models. |
GPT-Fathom: Benchmarking Large Language Models to Decipher the Evolutionary Path towards GPT-4 and Beyond (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing LLM leaderboards often reference scores reported in other papers without consistent settings and prompts, which may encourage cherry-picking favored settings and for better results. |
| Approach: | They propose an open-source and reproducible LLM evaluation suite built on top of OpenAI Evals that systematically evaluates 10+ leading LLMs and OpenAI’s legacy models on 20+ curated benchmarks across 7 capability categories. |
| Outcome: | The evaluation suite is built on top of OpenAI Evals and evaluates 10+ leading LLMs and OpenAI’s legacy models on 20+ curated benchmarks across 7 capability categories. |
LAraBench: Benchmarking Arabic AI with Large Language Models (2024.eacl-long)
Copied to clipboard
Ahmed Abdelali, Hamdy Mubarak, Shammur Chowdhury, Maram Hasanain, Basel Mousi, Sabri Boughorbel, Samir Abdaljalil, Yassine El Kheir, Daniel Izham, Fahim Dalvi, Majd Hawasly, Nizi Nazar, Youssef Elshahawy, Ahmed Ali, Nadir Durrani, Natasa Milic-Frayling, Firoj Alam
| Challenge: | Recent advances in Large Language Models (LLMs) have significantly influenced the landscape of language and speech research. |
| Approach: | They used GPT-3.5-turbo, GPT-4, BLOOMZ, Jais-13b-chat, Whisper, and USM to tackle 33 distinct tasks across 61 datasets. |
| Outcome: | The proposed model outperforms SOTA models in zero-shot learning, with a few exceptions. |
Unveiling the Generalization Power of Fine-Tuned Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated exceptional multitasking abilities, but the comprehensive effects of fine-tuning on the LLMs’ generalization ability are not fully understood. |
| Approach: | They conduct extensive experiments across five distinct language tasks on different datasets to investigate whether fine-tuning affects the generalization ability intrinsic to LLMs. |
| Outcome: | The proposed model can generalize to different domains and tasks by integrating the in-context learning strategy during fine-tuning on generation tasks. |