Thank You, Stingray: Multilingual Large Language Models Can Not (Yet) Disambiguate Cross-Lingual Word Senses (2025.findings-naacl)
Copied to clipboard
Samuel Cahyawijaya, Ruochen Zhang, Jan Christian Blaise Cruz, Holy Lovenia, Elisa Gilbert, Hiroki Nomoto, Alham Fikri Aji
| Challenge: | Existing studies on multilingual large language models have raised concerns about their reliability beyond English. |
| Approach: | They propose a benchmark for cross-lingual sense disambiguation that uses false friends to identify the limitation of cross-linguistic sense disembarrassment in LLMs. |
| Outcome: | The proposed benchmark pinpoints the limitation of cross-lingual sense disambiguation in LLMs by using false friends in four languages. |
Similar Papers
MuBench: Assessment of Multilingual Capabilities of Large Language Models Across 61 Languages (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation datasets lack cross-lingual alignment, leaving assessments of multilingual capabilities fragmented in both language and skill coverage. |
| Approach: | They propose to use multilingual consistency as a complementary metric to assess performance bottlenecks and guide model improvement. |
| Outcome: | The proposed model lacks cross-lingual alignment and language coverage gaps between state-of-the-art models. |
Cross-Lingual Pitfalls: Automatic Probing Cross-Lingual Weakness of Multilingual Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models have achieved remarkable success in Natural Language Processing, yet their cross-lingual consistency remains a significant challenge. |
| Approach: | They propose a method to identify cross-lingual weaknesses in Large Language Models . they construct bilingual question pairs that expose performance discrepancies between English and target languages . |
| Outcome: | The proposed method uncovers over 50% accuracy drops in target languages across models. |
MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation (2025.emnlp-main)
Copied to clipboard
Weihao Xuan, Rui Yang, Heli Qi, Qingcheng Zeng, Yunze Xiao, Aosong Feng, Dairui Liu, Yun Xing, Junjue Wang, Fan Gao, Jinghui Lu, Yuang Jiang, Huitao Li, Xin Li, Kunyu Yu, Ruihai Dong, Shangding Gu, Yuekang Li, Xiaofei Xie, Felix Juefei-Xu, Foutse Khomh, Osamu Yoshie, Qingyu Chen, Douglas Teodoro, Nan Liu, Randy Goebel, Lei Ma, Edison Marrese-Taylor, Shijian Lu, Yusuke Iwasawa, Yutaka Matsuo, Irene Li
| Challenge: | Existing large language model evaluation benchmarks focus on English, while current multilingual tasks lack parallel questions that specifically assess cross-lingual reasoning abilities. |
| Approach: | They propose a comprehensive benchmark covering 29 languages, built on an English benchmark. |
| Outcome: | The MMLU-ProX is a comprehensive benchmark covering 29 languages, built on an English benchmark. |
7 Points to Tsinghua but 10 Points to ? Assessing Large Language Models in Agentic Multilingual National Bias (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models have garnered significant attention for their capabilities in multilingual natural language processing, but studies on risks associated with cross biases are limited to immediate context preferences. |
| Approach: | They investigate multilingual bias in state-of-the-art Large Language Models by analyzing their responses to decision-making tasks across multiple languages. |
| Outcome: | The proposed model can provide personalized advice across university applications, travel, and relocation scenarios. |
LLM-XTM: Enhancing Cross-Lingual Topic Models with Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing cross-lingual topic models depend on sparse bilingual resources and often yield incoherent or weakly aligned topics. |
| Approach: | They propose a framework that integrates LLM-guided topic refinement with self-consistency uncertainty quantification to enable black-box, stable, and scalable enhancement of cross-lingual topic models. |
| Outcome: | Experiments on multilingual corpora show that the proposed framework achieves superior topic coherence and alignment while reducing reliance on bilingual dictionaries and expensive LLM calls. |
MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment (2025.findings-acl)
Copied to clipboard
Amir Hossein Kargaran, Ali Modarressi, Nafiseh Nikeghbal, Jana Diesner, François Yvon, Hinrich Schuetze
| Challenge: | Existing benchmarks for multilinguality for English-centric large language models focus on classic tasks or cover a minimal number of languages. |
| Approach: | They propose a method to assess multilingual capabilities of pre-trained LLMs using parallel sentences. |
| Outcome: | The proposed method evaluates the multilingual capabilities of pre-trained English-centric models using parallel sentences. |
SANDWiCH: Semantical Analysis of Neighbours for Disambiguating Words in Context ad Hoc (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent studies show that language understanding offered by chat-based Large Language Models is limited and far from human-like performance. |
| Approach: | They propose a framework for multilingual Word Sense Disambiguation using group algebra. |
| Outcome: | The proposed framework surpasses the performance of current alternatives even in low-resource languages while reducing the parameter count by 72%. |
Multilingual Large Language Models Are Not (Yet) Code-Switchers (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing multilingual Large Language Models are not specifically trained with objectives for managing code-switching scenarios. |
| Approach: | They propose to use multilingual Large Language Models to perform sentiment analysis, machine translation, summarization and word-level language identification to compare their performance to fine-tuned models of much smaller scales. |
| Outcome: | The proposed models show that they underperform in comparison to fine-tuned models of much smaller scales. |
1+1>2: Can Large Language Models Serve as Cross-Lingual Knowledge Aggregators? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been recognized for their impressive capabilities in natural language processing (NLP). |
| Approach: | They propose a method to enhance the multilingual performance of Large Language Models by aggregating knowledge from diverse languages. |
| Outcome: | The proposed method reduces the performance disparity across languages and offers valuable insights for further exploration. |
MultiLexBATS: Multilingual Dataset of Lexical Semantic Relations (2024.lrec-main)
Copied to clipboard
Dagmar Gromann, Hugo Goncalo Oliveira, Lucia Pitarch, Elena-Simona Apostol, Jordi Bernad, Eliot Bytyçi, Chiara Cantone, Sara Carvalho, Francesca Frontini, Radovan Garabik, Jorge Gracia, Letizia Granata, Fahad Khan, Timotej Knez, Penny Labropoulou, Chaya Liebeskind, Maria Pia Di Buono, Ana Ostroški Anić, Sigita Rackevičienė, Ricardo Rodrigues, Gilles Sérasset, Linas Selmistraitis, Mahammadou Sidibé, Purificação Silvano, Blerina Spahiu, Enriketa Sogutlu, Ranka Stanković, Ciprian-Octavian Truică, Giedre Valunaite Oleskeviciene, Slavko Zitnik, Katerina Zdravkova
| Challenge: | Prior work has focused on analysing lexical semantic relations in word embeddings or probing pretrained language models (PLMs) with some exceptions. |
| Approach: | They propose to use a multilingual parallel dataset of lexical semantic relations adapted from BATS in 15 languages including low-resource languages such as Bambara, Lithuanian, and Albanian as an experiment on cross-lingual transfer of relational knowledge. |
| Outcome: | The proposed dataset is adapted from a BATS-based dataset in 15 languages including low-resource languages such as Bambara, Lithuanian, and Albanian. |