Tyler Chang, Kishaloy Halder, Neha Anna John, Yogarshi Vyas, Yassine Benajiba, Miguel Ballesteros, Dan Roth
| Challenge: | Existing metrics for dataset drift have not considered specific dimensions of linguistic drift that affect model performance. |
| Approach: | They propose three dimensions of linguistic dataset drift: vocabulary, structural, and semantic drift. |
| Outcome: | The proposed metrics are more effective than previous metrics at predicting out-of-domain model accuracies compared to popular fine-tuned embedding distances . |
Similar Papers
Domain Divergences: A Survey and Empirical Analysis (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing literature on divergence measures is lacking in predicting performance of models in new domains. |
| Approach: | They propose a taxonomy of divergence measures consisting of three classes — Information-theoretic, Geometric, and Higher-order measures and identify the relationships between them. |
| Outcome: | The proposed measures are based on three novel use-cases and identify that they are prevalent in three domains and higher-order measures are more common in two. |
A Tutorial on Evaluation Metrics used in Natural Language Generation (2021.naacl-tutorials)
Copied to clipboard
| Challenge: | This tutorial presents the evolution of automatic evaluation metrics to their current state along with emerging trends in this field. |
| Approach: | This tutorial presents the evolution of automatic evaluation metrics to their current state . it aims to assess the extent of scientific progress made and identify areas/components that need improvement . |
| Outcome: | This tutorial presents the evolution of automatic evaluation metrics to their current state along with emerging trends in this field. |
Semantic Shift Stability: Efficient Way to Detect Performance Degradation of Word Embeddings and Pre-trained Language Models (2022.aacl-main)
Copied to clipboard
| Challenge: | Existing methods to detect time-series performance degradation of word embeddings and pre-trained language models are not efficient. |
| Approach: | They propose a way to detect time-series performance degradation by calculating the degree of semantic shift. |
| Outcome: | The proposed method detects time-series performance degradation in Japanese and English datasets. |
Evaluating Text Style Transfer Evaluation: Are There Any Reliable Metrics? (2025.naacl-srw)
Copied to clipboard
| Challenge: | Text style transfer (TST) is a multidimensional task requiring the assessment of style transfer accuracy, content preservation, and naturalness. |
| Approach: | They propose to use text style transfer metrics to evaluate outputs of text editors . they also investigate the potential of large language models as tools for TST evaluation . |
| Outcome: | The proposed methods provide better insights than existing metrics, the authors show . their meta-evaluation through correlation with hu-man judgments shows they are effective . |
Current Semantic-change Quantification Methods Struggle with Semantic Change Discovery in the Wild (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for lexical semantic-change detection quantify changes in the meaning of words over time. |
| Approach: | They propose to use a top-k setup to evaluate semantic-change discovery despite lacking complete annotations on a battery of semantic-changing detection methods. |
| Outcome: | The proposed setup extends the annotations in the commonly used LiverpoolFC and SemEval-EN benchmarks by 85% and 90%. |
On the Effect of Hyperparameters in Language Modeling for Computational Linguistics (2026.acl-long)
Copied to clipboard
| Challenge: | Training language models and examining their linguistic behaviors is a common protocol in computational linguistics for studying linguistic phenomena and modeling human language processing. |
| Approach: | They replicate three prior studies with hyperparameters varied within a practical range and show that modest hyperparametric changes can alter qualitative conclusions about models’ linguistic abilities. |
| Outcome: | The results show that hyperparameter changes can alter qualitative conclusions and reverse the ranking of models. |
Explaining and Improving BERT Performance on Lexical Semantic Change Detection (2021.eacl-srw)
Copied to clipboard
| Challenge: | Lexical semantic change detection is still a challenging field due to the success of type-based embeddings in SemEval-2020 Task 1 and other NLP tasks. |
| Approach: | They compare the performance of BERT embeddings with results from the word sense disambiguation dataset underlying SemEval-2020 Task 1 and the Italian follow-up task DIACR-Ita. |
| Outcome: | The proposed model outperforms token-based embeddings on lexical semantic change detection tasks. |
Uncertainty in Language Models: Assessment through Rank-Calibration (2024.emnlp-main)
Copied to clipboard
Xinmeng Huang, Shuo Li, Mengxin Yu, Matteo Sesia, Hamed Hassani, Insup Lee, Osbert Bastani, Edgar Dobriban
| Challenge: | Language Models (LMs) have shown promising performance in natural language generation . however, it is crucial to correctly quantify their level of uncertainty in responding to inputs. |
| Approach: | They propose a framework to quantify uncertainty and confidence for Large Language Models . they use a Rank-calibration framework to measure uncertainty and confident responses . |
| Outcome: | The proposed framework assesses uncertainty and confidence measures for LMs. |
Quantifying Language Disparities in Multilingual Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Contemporary NLP development relies on digital language datasets to build large language models. |
| Approach: | They propose a framework that disentangles confounding variables and introduces interpretable metrics to quantify model performance and language disparities. |
| Outcome: | The proposed framework provides a more reliable measurement of model performance and language disparities for low-resource languages. |
Analysing Lexical Semantic Change with Contextualised Word Representations (2020.acl-main)
Copied to clipboard
| Challenge: | Existing studies on lexical semantic change have focused on detecting and characterising word meaning shifts using distributional semantic models. |
| Approach: | They propose a method that exploits the BERT neural language model to obtain representations of word usages, clusters these representations into usage types, and measures change along time with three proposed metrics. |
| Outcome: | The proposed method captures a variety of synchronic and diachronic linguistic phenomena and is highly reproducible and reproducible. |