ReTRE: Benchmarking LLM Transfer Robustness with Structure-Preserving Variants (2026.acl-long)
Copied to clipboard
| Challenge: | Learning transfer theory emphasizes that applying acquired knowledge to novel manifestations is a key signal of deep understanding |
| Approach: | They propose a benchmark that probes transfer robustness along two rewrite levels: Near Transfer and Far Transfer. |
| Outcome: | The proposed benchmark demonstrates that large language models are robust when faced with novel manifestations of the same problem. |
Similar Papers
SCORE: Systematic COnsistency and Robustness Evaluation for Large Language Models (2025.naacl-industry)
Copied to clipboard
| Challenge: | Typical evaluations of Large Language Models (LLMs) report a single accuracy metric per dataset, often derived from an optimized setup. |
| Approach: | They propose a framework for non-adversarial evaluation of large language models that evaluates models by repeatedly testing them on the same benchmarks in various setups. |
| Outcome: | The proposed framework evaluates models by repeatedly testing them on the same benchmarks in various setups to give a realistic estimate of their accuracy and consistency. |
Improving the OOD Performance of Closed-Source LLMs on NLI Through Strategic Data Selection (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods to improve robustness require changing the fine-tuning process or large-scale data augmentation, which are infeasible or cost prohibitive for closed-source models. |
| Approach: | They propose to prioritize more complex examples or replace existing training examples with LLM-generated data to improve performance on OOD NLI datasets. |
| Outcome: | The proposed methods improve performance on difficult OOD datasets while training with synthetic data leads to substantial improvements on easier OOD data. |
Ready to Translate, Not to Represent? Bias and Performance Gaps in Multilingual LLMs Across Language Families and Domains (2026.findings-acl)
Copied to clipboard
Md. Faiyaz Abdullah Sayeedi, Subhey Sadi Rahman, Md. Mahbub Alam, Md. Adnanul Islam, Jannatul Ferdous Deepti, Tasnim Mohiuddin, Md Mofijul Islam, Swakkhar Shatabda
| Challenge: | Large Language Models (LLMs) have redefined Machine Translation, enabling context-aware and fluent translations across hundreds of languages and textual domains. |
| Approach: | They propose a framework and dataset to evaluate the translation quality and fairness of open-source LLMs. |
| Outcome: | The proposed framework and dataset evaluates translation quality and fairness of open-source LLMs. |
How to Improve LLMs’ Performance on Specific Languages: A Perspective on LLM-Derived Language Similarity (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) exhibit uneven performance across languages. |
| Approach: | They propose to use a framework to quantify the similarity within each language pair through both the lenses of language-specific performance patterns and cross-lingual transferability. |
| Outcome: | The proposed approach outperforms traditional linguistic typology and cross-lingual transferability measures on multilingual LLMs. |
Bridging Distribution Gap via Semantic Rewriting with LLMs to Enhance OOD Robustness (2024.acl-srw)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning on indistribution data fail to provide robustness against distribution shifts limiting the practical deployment of LLMs in dynamic real-world scenarios. |
| Approach: | They propose a method that leverages the flexibility of LLMs to align both in-distribution (ID) and OOD data with the LLM's distributions. |
| Outcome: | The proposed method outperforms fine-tuning methods on OOD tasks and benchmark datasets. |
Close or Cloze? Assessing the Robustness of Large Language Models to Adversarial Perturbations via Word Recovery (2025.coling-main)
Copied to clipboard
| Challenge: | Existing models implicitly recover the original text, but it is unclear when they rely on context and when they implicitly do so. |
| Approach: | They propose to use a dictionary to recover adversarial words by using a phonetic, typo, and visual attack to study word recovery performance. |
| Outcome: | The proposed model outperforms open-source models on hateful, offensive, and toxic classification tasks. |
Contextual Refinement of Translations: Large Language Models for Sentence and Document-Level Post-Editing (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models have demonstrated considerable success in various natural language processing tasks, but their performance in NMT tasks is still underexplored. |
| Approach: | They propose to use LLMs as automatic post-editors rather than direct translators to improve BLEU and COMET performance. |
| Outcome: | The proposed approach improves BLEU but COMET performance compared to in-context learning. |
A Novel Metric for Measuring the Robustness of Large Language Models in Non-adversarial Scenarios (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Using large language models, we evaluated their robustness on multiple datasets. |
| Approach: | They propose a new metric for assessing model robustness by empirical evaluation of several models on multiple datasets. |
| Outcome: | The proposed metric is based on a set of datasets that are constructed by introducing naturally-occurring, non-malicious perturbations or by generating semantically equivalent paraphrases of input questions or statements. |
Methods for Estimating and Improving Robustness of Language Models (2022.naacl-srw)
Copied to clipboard
| Challenge: | Large language models suffer from weak generalisation ability due to shallow textual relations over full semantic complexity of the problem. |
| Approach: | They propose to incorporate some of these measures into training objectives to enhance distributional robustness of LLMs. |
| Outcome: | The proposed models outperform human models on complex tasks and outperformed other models on deep networks. |
Harnessing Consistency for Robust Test-Time LLM Ensemble (2026.findings-eacl)
Copied to clipboard
Zhichen Zeng, Qi Yu, Xiao Lin, Ruizhong Qiu, Xuying Ning, Tianxin Wei, Yuchen Yan, Jingrui He, Hanghang Tong
| Challenge: | Existing efforts to improve LLM ensemble quality have focused on model consistency, but failures are often due to heterogeneous tokenization schemes and varying model expertise. |
| Approach: | They propose a plug-and-play technique that harnesses model consistency for robust LLM ensemble. |
| Outcome: | The proposed technique improves ensemble performance and robustness against erroneous signals. |