MaXIFE: Multilingual and Cross-lingual Instruction Following Evaluation (2025.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation methods focus on single-language scenarios, overlooking multilingual and cross-lingual contexts. |
| Approach: | They propose a tool to assess instruction-following capabilities across 23 different languages with 1667 verifiable instruction tasks. |
| Outcome: | MaXIFE evaluates instruction-following capabilities across 23 languages with 1667 verifiable instruction tasks. |
Similar Papers
M-IFEval: Multilingual Instruction-Following Evaluation (2025.findings-naacl)
Copied to clipboard
| Challenge: | Instruction following is a core capability of Large language models (LLMs), making evaluating this capability essential to understanding these models. |
| Approach: | They propose a multilingual instruction following evaluation benchmark that expands to other languages . they propose to use both general and language-specific instructions to evaluate LLMs . |
| Outcome: | The proposed benchmark is extended to French, Japanese, and Spanish . it shows that performance across languages and instruction types can vary widely . |
ReIFE: Re-evaluating Instruction-Following Evaluation (2025.naacl-long)
Copied to clipboard
Yixin Liu, Kejian Shi, Alexander Fabbri, Yilun Zhao, PeiFeng Wang, Chien-Sheng Wu, Shafiq Joty, Arman Cohan
| Challenge: | Existing evaluations of large language models (LLMs) for instruction following are incomplete. |
| Approach: | They propose to use 25 base LLMs and 15 recently proposed evaluation protocols to evaluate instruction following on 4 human-annotated datasets. |
| Outcome: | The proposed evaluations identify the best-performing base LLMs and evaluation protocols with a high degree of robustness. |
BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing multilingual benchmarks focus primarily on language understanding tasks. |
| Approach: | They develop a multi-way multilingual benchmark that measures critical capabilities of large language models across languages. |
| Outcome: | Extensive experiments on BenchMAX reveal uneven utilization of core capabilities across languages, emphasizing the performance gaps that scaling model size alone does not resolve. |
Marco-Bench-MIF: On Multilingual Instruction-Following Capability of Large Language (2025.acl-long)
Copied to clipboard
Bo Zeng, Chenyang Lyu, Sinuo Liu, Mingyan Zeng, Minghao Wu, Xuanfan Ni, Tianqi Shi, Yu Zhao, Yefeng Liu, Chenyu Zhu, Ruizhe Li, Jiahui Geng, Qing Li, Yu Tong, Longyue Wang, Weihua Luo, Kaifu Zhang
| Challenge: | Existing datasets for instruction-following are monolingual and centered on English . existing data are unable to capture linguistic and cultural subtle differences . |
| Approach: | They propose an extension of IFEval to a localized multilingual version called Marco-Bench-MIF . their benchmark addresses linguistic constraints and cultural references via translation and verification . |
| Outcome: | The proposed extension of IFEval to a localized multilingual version covers 30 languages with varying levels of localization. |
Multi-lingual Functional Evaluation for Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Multilingual competence in large language models is often evaluated via static data benchmarks such as Belebele, M-MMLU and M-GSM. |
| Approach: | They extend existing functional benchmark templates from English to five additional languages that span the range of resources available for NLP: French, Spanish, Hindi, Arabic and Yoruba. |
| Outcome: | The proposed models are translated from English to French, Spanish, Hindi, Arabic and Yoruba. |
MuBench: Assessment of Multilingual Capabilities of Large Language Models Across 61 Languages (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation datasets lack cross-lingual alignment, leaving assessments of multilingual capabilities fragmented in both language and skill coverage. |
| Approach: | They propose to use multilingual consistency as a complementary metric to assess performance bottlenecks and guide model improvement. |
| Outcome: | The proposed model lacks cross-lingual alignment and language coverage gaps between state-of-the-art models. |
MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation (2025.emnlp-main)
Copied to clipboard
Weihao Xuan, Rui Yang, Heli Qi, Qingcheng Zeng, Yunze Xiao, Aosong Feng, Dairui Liu, Yun Xing, Junjue Wang, Fan Gao, Jinghui Lu, Yuang Jiang, Huitao Li, Xin Li, Kunyu Yu, Ruihai Dong, Shangding Gu, Yuekang Li, Xiaofei Xie, Felix Juefei-Xu, Foutse Khomh, Osamu Yoshie, Qingyu Chen, Douglas Teodoro, Nan Liu, Randy Goebel, Lei Ma, Edison Marrese-Taylor, Shijian Lu, Yusuke Iwasawa, Yutaka Matsuo, Irene Li
| Challenge: | Existing large language model evaluation benchmarks focus on English, while current multilingual tasks lack parallel questions that specifically assess cross-lingual reasoning abilities. |
| Approach: | They propose a comprehensive benchmark covering 29 languages, built on an English benchmark. |
| Outcome: | The MMLU-ProX is a comprehensive benchmark covering 29 languages, built on an English benchmark. |
LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedback (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent multilingual models support limited number of human languages due to lack of training data for low resource languages. |
| Approach: | They propose a multilingual multilingual LLM that scales to 100 languages . they use a human feedback dataset and a data set to perform multilingual instruction tuning . |
| Outcome: | The proposed model outperforms its peers on five multilingual benchmarks. |
The SIFo Benchmark: Investigating the Sequential Instruction Following Ability of Large Language Models (2024.findings-emnlp)
Copied to clipboard
Xinyi Chen, Baohao Liao, Jirui Qi, Panagiotis Eustratiadis, Christof Monz, Arianna Bisazza, Maarten Rijke
| Challenge: | Current evaluation resources for instruction following focus on single task instructions, but the instruction sequences in these benchmarks often lack coherence. |
| Approach: | They propose to evaluate models’ abilities to follow multiple instructions through sequential instruction following tasks using four tasks to assess different aspects of sequential instruction followed. |
| Outcome: | The proposed benchmark outperforms open-source and closed-source models on four tasks assessing different aspects of sequential instruction following. |
EIFFEL: a novel benchmark to measure bias of English heavy training on French idiomatic expressions (2026.acl-long)
Copied to clipboard
| Challenge: | Mainstream multilingual models are generally trained on a much higher proportion of English data . this raises questions about their ability to capture linguistic features specific to non-English languages . |
| Approach: | They propose a benchmark to test multilingual LLMs' ability to capture linguistic features in other languages. |
| Outcome: | The proposed benchmark shows that multilingual models can capture features in non-English languages and cultural norms. |