| Challenge: | Performance drop due to domain-shift is an endemic problem for NLP models in production. |
| Approach: | They propose to use H-divergence, reverse classification accuracy and confidence measures to predict performance drop under domain-shift without any target domain labels. |
| Outcome: | The proposed method predicts performance drops with an error rate as low as 2.15% and 0.89% for sentiment analysis and POS tagging respectively. |
Similar Papers
Domain Divergences: A Survey and Empirical Analysis (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing literature on divergence measures is lacking in predicting performance of models in new domains. |
| Approach: | They propose a taxonomy of divergence measures consisting of three classes — Information-theoretic, Geometric, and Higher-order measures and identify the relationships between them. |
| Outcome: | The proposed measures are based on three novel use-cases and identify that they are prevalent in three domains and higher-order measures are more common in two. |
Measuring the Robustness of NLP Models to Domain Shifts (2024.findings-emnlp)
Copied to clipboard
Nitay Calderon, Naveh Porat, Eyal Ben-David, Alexander Chapanin, Zorik Gekhman, Nadav Oved, Vitaly Shalumov, Roi Reichart
| Challenge: | Existing research on domain robustness (DR) relies on the Source Drop (SD) but lacks a complementary metric, a new study finds . |
| Approach: | They propose to use the Target Drop (TD) to measure domain DR . they use a DR benchmark consisting of 7 diverse tasks to measure both metrics . |
| Outcome: | The proposed model types excel in-domain, but few-shot LLMs often surpass them cross-domain showing better robustness. |
DA-Pred: Performance Prediction for Text Summarization under Domain-Shift and Instruct-Tuning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) often don’t perform as expected under Domain Shift or after Instruct-tuning. |
| Approach: | They propose a method that uses the known performance in high-resource domains and fine-tuning settings to predict performance in low-resourced domains or base models. |
| Outcome: | The proposed method can help researchers decide if resources should be allocated for data labeling and LLM Instruct-tuning. |
Predicting Performance for Natural Language Processing Tasks (2020.acl-main)
Copied to clipboard
| Challenge: | Natural language processing (NLP) is a vast field, with a wide variety of tasks, languages, and domains. |
| Approach: | They build regression models to predict evaluation score of an NLP experiment . they find that their models can produce meaningful predictions over unseen languages . |
| Outcome: | The proposed model outperforms baseline models and human experts on 9 different tasks. |
Strong Baselines for Neural Semi-Supervised Learning under Domain Shift (P18-1)
Copied to clipboard
| Challenge: | Existing neural models for learning under domain shifts only evaluate on a single task, on proprietary datasets, or compare to weak baselines. |
| Approach: | They propose a multi-task tri-training method that reduces time and space complexity of classic bootstrapping approaches. |
| Outcome: | The proposed method outperforms the state-of-the-art for sentiment analysis on two benchmarks. |
How Bad are PoS Tagger in Cross-Corpora Settings? Evaluating Annotation Divergence in the UD Project. (N19-1)
Copied to clipboard
| Challenge: | Using annotation variation principles, Part-of-Speech tagging performance degrades when applied to test sentences that depart from training data. |
| Approach: | They propose to use the annotation variation principle to identify inconsistencies between annotations . they also evaluate their impact on prediction performance . |
| Outcome: | The proposed method can detect errors in gold standard annotations and improve prediction performance. |
Evaluating Models’ Local Decision Boundaries via Contrast Sets (2020.findings-emnlp)
Copied to clipboard
Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, Ben Zhou
| Challenge: | Standard test sets for supervised learning evaluate in-distribution generalization but are misleading when a dataset has systematic gaps. |
| Approach: | They propose a more rigorous annotation paradigm for NLP that helps to close systematic gaps in the test data. |
| Outcome: | The proposed model performs significantly lower on contrast sets than on the original test sets—up to 25% in some cases. |
Neural Unsupervised Domain Adaptation in NLP—A Survey (2020.coling-main)
Copied to clipboard
| Challenge: | Deep neural networks excel at learning from labeled data, but learning from unlabeled data remains a challenge. |
| Approach: | They review neural unsupervised domain adaptation techniques which do not require labeled target domain data. |
| Outcome: | The proposed techniques are more challenging yet widely applicable. |
Semantic Shift Stability: Efficient Way to Detect Performance Degradation of Word Embeddings and Pre-trained Language Models (2022.aacl-main)
Copied to clipboard
| Challenge: | Existing methods to detect time-series performance degradation of word embeddings and pre-trained language models are not efficient. |
| Approach: | They propose a way to detect time-series performance degradation by calculating the degree of semantic shift. |
| Outcome: | The proposed method detects time-series performance degradation in Japanese and English datasets. |
Unveiling the Multi-Annotation Process: Examining the Influence of Annotation Quantity and Instance Difficulty on Model Performance (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have shown that multi-annotator datasets can improve performance when they expand from a single annotation per instance to multiple annotations. |
| Approach: | They propose a multi-annotator simulation process to generate datasets with varying annotation budgets and compare them to a single annotation per instance. |
| Outcome: | The proposed model can generate datasets with varying annotation budgets and show that similar datasets can lead to varying performance gains. |