Generalized Entropy Regularization or: There’s Nothing Special about Label Smoothing (2020.acl-main)
Copied to clipboard
| Challenge: | Prior work has explored regularizing the output distributions of probabilistic models to alleviate overfitting. |
| Approach: | They propose a family of entropy regularizers that have a connection to regularization . they find that label smoothing provably does not allow for sparsity in an output distribution . |
| Outcome: | The proposed method improves the relationship between model entropy and performance on language generation tasks. |
Similar Papers
Towards a Better Understanding of Label Smoothing in Neural Machine Translation (2020.aacl-main)
Copied to clipboard
| Challenge: | In recent years, Neural Network (NN) models bring steady and concrete improvements on the task of Machine Translation (MT). |
| Approach: | They propose to penalize over-confident outputs and regularize the model so that its outputs do not diverge too much from some prior distribution. |
| Outcome: | The proposed method is well-motivated and can improve the performance of strong neural machine translation systems. |
The Role of n-gram Smoothing in the Age of Neural Networks (2024.naacl-long)
Copied to clipboard
| Challenge: | n-gram smoothing techniques were used to overcome overfitting problems in neural language models for decades. |
| Approach: | They propose to convert any n-gram smoothing technique into a regularizer compatible with neural language models. |
| Outcome: | The proposed regularizers outperform label smoothing on language modeling and machine translation. |
Adaptive Label Smoothing with Self-Knowledge in Natural Language Generation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Overconfidence in model generalization and calibration has been shown to impair model generalisation and calibration. |
| Approach: | They propose a regularization scheme that takes model probability into account and takes it into account . they use a prior label distribution to smooth target labels . |
| Outcome: | The proposed model improves model generalization and calibration by taking model probability into account. |
Large Language Models Do Multi-Label Classification Differently (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multi-label classification is prevalent in real-world settings, but the behavior of Large Language Models (LLMs) in this setting is understudied. |
| Approach: | They propose to use initial probability distributions to analyze output distributions of LLMs at each label generation step to find out how LLM models perform multi-label classification. |
| Outcome: | The proposed methods improve alignment and predictive performance over existing methods. |
Systematic Generalization in Language Models Scales with Information Entropy (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for assessing compositional behavior are unclear on how to measure the difficulty of a systematic generalization problem. |
| Approach: | They propose a framework for measuring entropy in a sequence-to-sequence task and a method for measuring it. |
| Outcome: | The proposed framework scales with the entropy of the distribution of component parts in the training data. |
Enhancing Language Model Alignment: A Confidence-Based Approach to Label Smoothing (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have remarkable capabilities across various domains . Reinforcement Learning with Human Feedback (RLHF) phase is crucial for training . label smoothing is a technique that replaces hard labels with soft labels . |
| Approach: | They propose a method that iteratively updates the label smoothing parameter based on preference labels and model forecasts. |
| Outcome: | The proposed method improves the performance of large language models on state-of-the-art alignment tasks. |
In and Out-of-Domain Text Adversarial Robustness via Label Smoothing (2023.acl-short)
Copied to clipboard
| Challenge: | Existing studies show that state-of-the-art NLP models are vulnerable to adversarial attacks . label smoothing has been proven effective in a variety of applications and modalities . |
| Approach: | They propose to use label smoothing to improve adversarial robustness in pre-trained models against various popular attacks. |
| Outcome: | The proposed method significantly improves adversarial robustness in pre-trained models against various popular attacks. |
Limitations of Autoregressive Models and Their Alternatives (2021.naacl-main)
Copied to clipboard
| Challenge: | Standard autoregressive language models only perform polynomial-time computation to compute probability of next symbol. |
| Approach: | authors propose alternative to standard autoregressive language models that use polynomial-time computation to compute probability of next symbol. |
| Outcome: | a large model size can grow superpolynomially in length, allowing it to store precomputed results and verify solutions. |
Out-of-Distribution Generalization in Natural Language Processing: Past, Present, and Future (2023.emnlp-main)
Copied to clipboard
Linyi Yang, Yaoxian Song, Xuan Ren, Chenyang Lyu, Yidong Wang, Jingming Zhuo, Lingqiao Liu, Jindong Wang, Jennifer Foster, Yue Zhang
| Challenge: | Existing literature on the generalization of machine learning models to out-of-distribution data is lacking. |
| Approach: | They propose to present the first comprehensive review of recent progress, methods, and evaluations on the generalization challenge from an OOD perspective in natural language understanding. |
| Outcome: | The proposed survey provides the first comprehensive review of recent progress, methods, and evaluations on the generalization challenge from an OOD perspective in natural language understanding. |
Unifying Input and Output Smoothing in Neural Machine Translation (2020.coling-main)
Copied to clipboard
| Challenge: | Recent methods that smooth input and output of neural machine translation systems bring significant improvements in performance. |
| Approach: | They propose a method that replaces one-hot representations with soft posterior distributions of an external language model, smoothing the input of machine translation systems. |
| Outcome: | The proposed method improves translation performance on small datasets and larger datasets. |