| Challenge: | Existing literature on divergence measures is lacking in predicting performance of models in new domains. |
| Approach: | They propose a taxonomy of divergence measures consisting of three classes — Information-theoretic, Geometric, and Higher-order measures and identify the relationships between them. |
| Outcome: | The proposed measures are based on three novel use-cases and identify that they are prevalent in three domains and higher-order measures are more common in two. |
Similar Papers
Divergence-Based Domain Transferability for Zero-Shot Classification (2023.findings-eacl)
Copied to clipboard
| Challenge: | a recent study shows that fine-tuning of neural models can improve performance on language-based tasks without brute-force searching effective task combinations. |
| Approach: | They propose to use divergence measures to estimate whether one task pair will perform better than another . they use 58 tasks and 6,600 task pair combinations to study the effect of different tuning methods . |
| Outcome: | The proposed method reduces end-to-end runtime by 40% by estimating transferability . the proposed method is based on 58 tasks and over 6,600 task pair combinations . |
Neural Unsupervised Domain Adaptation in NLP—A Survey (2020.coling-main)
Copied to clipboard
| Challenge: | Deep neural networks excel at learning from labeled data, but learning from unlabeled data remains a challenge. |
| Approach: | They review neural unsupervised domain adaptation techniques which do not require labeled target domain data. |
| Outcome: | The proposed techniques are more challenging yet widely applicable. |
We Need to Measure Data Diversity in NLP — Better and Broader (2025.emnlp-main)
Copied to clipboard
| Challenge: | Language models exhibit remarkable natural language understanding and generation capabilities, but they have serious flaws, such as societal biases and spurious correlations. |
| Approach: | They argue that interdisciplinary perspectives are essential for developing more fine-grained and valid measures of data diversity. |
| Outcome: | The proposed measures are based on interdisciplinary perspectives and include a variety of datasets. |
Measure and Improve Robustness in NLP Models: A Survey (2022.naacl-main)
Copied to clipboard
| Challenge: | Despite the performance gains, NLP models are still fragile and brittle to out-of-domain data, adversarial attacks, or small perturbation to the input. |
| Approach: | They propose a survey of how to define, measure and improve robustness in NLP by connecting multiple definitions of robustness and identifying failures. |
| Outcome: | The proposed models are robust against unseen or challenging scenarios, but are still fragile and brittle to out-of-domain data and adversarial attacks. |
To Annotate or Not? Predicting Performance Drop under Domain Shift (D19-1)
Copied to clipboard
| Challenge: | Performance drop due to domain-shift is an endemic problem for NLP models in production. |
| Approach: | They propose to use H-divergence, reverse classification accuracy and confidence measures to predict performance drop under domain-shift without any target domain labels. |
| Outcome: | The proposed method predicts performance drops with an error rate as low as 2.15% and 0.89% for sentiment analysis and POS tagging respectively. |
Efficient Methods for Natural Language Processing: A Survey (2023.tacl-1)
Copied to clipboard
Marcos Treviso, Ji-Ung Lee, Tianchu Ji, Betty van Aken, Qingqing Cao, Manuel R. Ciosici, Michael Hassid, Kenneth Heafield, Sara Hooker, Colin Raffel, Pedro H. Martins, André F. T. Martins, Jessica Zosa Forde, Peter Milder, Edwin Simpson, Noam Slonim, Jesse Dodge, Emma Strubell, Niranjan Balasubramanian, Leon Derczynski, Iryna Gurevych, Roy Schwartz
| Challenge: | Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data, but using only scale to improve performance means resource consumption also grows. |
| Approach: | They propose to use data, time, storage, or energy to improve model performance. |
| Outcome: | The proposed methods and findings provide guidance for conducting NLP under limited resources and point towards promising research directions for developing more efficient methods. |
Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challenges (2025.acl-long)
Copied to clipboard
Bolei Ma, Yuting Li, Wei Zhou, Ziwei Gong, Yang Janet Liu, Katja Jasinskaja, Annemarie Friedrich, Julia Hirschberg, Frauke Kreuter, Barbara Plank
| Challenge: | linguistics studies how context influences meaning of language and how people use it to convey implied meanings, emotions, and intentions. |
| Approach: | They analyze task designs, data collection methods, evaluation approaches and their relevance to real-world applications. |
| Outcome: | The findings highlight emerging trends, challenges, and gaps in existing benchmarks . the findings will contribute to more nuanced and context-aware NLP models . |
A Tutorial on Evaluation Metrics used in Natural Language Generation (2021.naacl-tutorials)
Copied to clipboard
| Challenge: | This tutorial presents the evolution of automatic evaluation metrics to their current state along with emerging trends in this field. |
| Approach: | This tutorial presents the evolution of automatic evaluation metrics to their current state . it aims to assess the extent of scientific progress made and identify areas/components that need improvement . |
| Outcome: | This tutorial presents the evolution of automatic evaluation metrics to their current state along with emerging trends in this field. |
A Survey of Methods for Addressing Class Imbalance in Deep-Learning Based Natural Language Processing (2023.eacl-main)
Copied to clipboard
| Challenge: | Developing methods to improve model performance in imbalanced data settings has been an active area for decades . |
| Approach: | They propose to use sampling, data augmentation, choice of loss function, staged learning, or model design to address class imbalance in NLP. |
| Outcome: | The proposed approaches are evaluated on a variety of NLP tasks or in the computer vision community. |
DHP Benchmark: Are LLMs Good NLG Evaluators? (2025.findings-naacl)
Copied to clipboard
Yicheng Wang, Jiayi Yuan, Yu-Neng Chuang, Zhuoer Wang, Yingchi Liu, Mark Cusick, Param Kulkarni, Zhengping Ji, Yasser Ibrahim, Xia Hu
| Challenge: | Large Language Models (LLMs) are increasingly serving as evaluators in Natural Language Generation (NLG) tasks. |
| Approach: | They propose a framework that measures the discernment of Large Language Models (LLMs) across diverse NLG tasks. |
| Outcome: | The proposed framework provides quantitative discernment scores for LLMs across four NLG tasks. |