Predicting generalization performance with correctness discriminators (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models estimate accuracy of models on unlabeled test data, but they hide their own uncertainty. |
| Approach: | They propose a model that establishes upper and lower bounds on the accuracy without requiring gold labels for the unseen data. |
| Outcome: | The proposed model establishes upper and lower bounds on accuracy without requiring gold labels for the unseen data. |
Similar Papers
A Universal Discriminator for Zero-Shot Generalization (2023.acl-long)
Copied to clipboard
| Challenge: | Generative modeling has been the dominant approach for large-scale pretraining and zeroshot generalization. |
| Approach: | They propose a discriminator that predicts whether a text sample comes from the true data distribution and which option has the highest probability of coming from the real data distribution. |
| Outcome: | The proposed discriminative approach outperforms GANs on a number of NLP tasks by 16.0%, 7.8%, and 11.5% respectively. |
Out-of-Distribution Generalization in Natural Language Processing: Past, Present, and Future (2023.emnlp-main)
Copied to clipboard
Linyi Yang, Yaoxian Song, Xuan Ren, Chenyang Lyu, Yidong Wang, Jingming Zhuo, Lingqiao Liu, Jindong Wang, Jennifer Foster, Yue Zhang
| Challenge: | Existing literature on the generalization of machine learning models to out-of-distribution data is lacking. |
| Approach: | They propose to present the first comprehensive review of recent progress, methods, and evaluations on the generalization challenge from an OOD perspective in natural language understanding. |
| Outcome: | The proposed survey provides the first comprehensive review of recent progress, methods, and evaluations on the generalization challenge from an OOD perspective in natural language understanding. |
Calibrated Interpretation: Confidence Estimation in Semantic Parsing (2023.tacl-1)
Copied to clipboard
| Challenge: | Sequence generation models are increasingly being used to translate natural language into programs . calibration of such models is a key component of safety, says aaron sagar . |
| Approach: | They investigate whether calibration of popular generation models varies across models and datasets . they find that calibration varies among models and data sets, and that it is important to include it in evaluations if it is included . |
| Outcome: | The calibration of popular generation models varies across models and datasets . the authors find that the accuracy of models is dependent on confidence . |
Generalizing Trust: Weak-to-Strong Trustworthiness in Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Recent studies have highlighted weak-to-strong generalization, where a strong model trained only on a weak model’s labels surpasses the weak model in task performance. |
| Approach: | They propose two fundamental fine-tuning strategies that leverage trustworthiness regularization during the fine-uning of the weak model and the weak-to-strong transfer to improve trustworthy. |
| Outcome: | The proposed models show that they can generalize robustness, fairness, and privacy better when trained on weak models than models trained on strong models. |
Calibrating Structured Output Predictors for Natural Language Processing (2020.acl-main)
Copied to clipboard
| Challenge: | Several modern machine-learning based NLP systems can provide a confidence score with their output predictions. |
| Approach: | They propose a general calibration scheme for output entities of interest in NLP applications that can be used to calibrate confidence scores. |
| Outcome: | The proposed calibration scheme outperforms current calibration techniques for Named Entity Recognition, Part-of-speech tagging and Question Answering systems. |
On the Importance of Delexicalization for Fact Verification (D19-1)
Copied to clipboard
| Challenge: | Neural networks (NNs) perform state-of-the-art (SOA) performance in many complex tasks. |
| Approach: | They investigate the importance that a model assigns to various aspects of data . they experiment with two strategies of masking to mitigate this dependence on lexicalized information . |
| Outcome: | The proposed model improves on the in-domain dataset by 10% compared to the fully lexicalized model. |
Platt-Bin: Efficient Posterior Calibrated Training for NLP Classifiers (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for posterior calibration return uncalibrated estimations of class posteriors, thus leading to poorer generalization. |
| Approach: | They propose an end-to-end trained calibrator that directly optimizes the objective while minimizing the difference between predicted and empirical posterior probabilities. |
| Outcome: | The proposed calibrator reduces calibration error and improves performance on benchmark NLP classification tasks. |
Modular and Parameter-Efficient Fine-Tuning for NLP Models (2022.emnlp-tutorials)
Copied to clipboard
| Challenge: | State-of-the-art language models in NLP perform best when fine-tuned even on small datasets. |
| Approach: | They provide an overview of parameter-efficient fine-tuning methods and highlight similarities and differences . they highlight benefits and usage scenarios of a neglected property of parameter efficient models . |
| Outcome: | This paper provides an overview of parameter-efficient fine-tuning methods . it highlights similarities and differences by presenting them in a unified view . |
Methods for Estimating and Improving Robustness of Language Models (2022.naacl-srw)
Copied to clipboard
| Challenge: | Large language models suffer from weak generalisation ability due to shallow textual relations over full semantic complexity of the problem. |
| Approach: | They propose to incorporate some of these measures into training objectives to enhance distributional robustness of LLMs. |
| Outcome: | The proposed models outperform human models on complex tasks and outperformed other models on deep networks. |
Noise Stability Regularization for Improving BERT Fine-tuning (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent studies show that fine-tuning pre-trained language models is unstable when there are only a small number of training samples available. |
| Approach: | They propose to use a method to regularize noise in deep nets to improve fine-tuning on NLP tasks. |
| Outcome: | The proposed method improves fine-tuning on natural language processing tasks by incorporating noise to the input and demonstrating generalizability and stability. |