Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generation (2022.findings-acl)
Copied to clipboard
| Challenge: | Current language generation models suffer from issues such as repetition, incoherence, and hallucinations . |
| Approach: | They propose to analyze exposure bias from an imitation learning perspective and prove it is a problem . they show that exposure bias leads to an accumulation of errors during generation . |
| Outcome: | The proposed model fails to capture errors during generation and poor generation quality. |
Similar Papers
Exposure Bias versus Self-Recovery: Are Distortions Really Incremental for Autoregressive Text Generation? (2021.emnlp-main)
Copied to clipboard
| Challenge: | Exposure bias is a central problem for auto-regressive language models (LM) it is believed that teacher forcing would cause test-time generation to be incrementally distorted due to the training-generation discrepancy. |
| Approach: | They propose to quantify the impact of exposure bias in quality, diversity, consistency and consistency by using ground-truth data prefixes instead of prefix generated by the model. |
| Outcome: | The proposed model performs better when the training-generation discrepancy is removed . the model is more robust and self-recovery ability is shown to counter exposure bias. |
Generalization in Generation: A closer look at Exposure Bias (D19-56)
Copied to clipboard
| Challenge: | Autoregressive generative models are often criticized for using ground-truth contexts at training time but generated ones at test time. |
| Approach: | They propose that generalization is the underlying property to address and propose unconditional generation as its fundamental benchmark. |
| Outcome: | The proposed model is generalized and can handle true and generated contexts. |
From n-gram to Attention: How Model Architectures Learn and Propagate Bias in Language Modeling (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Current research on bias in language models focuses on data quality, not temporal influences of data. |
| Approach: | They propose a methodology to interpret the interaction between training data and model architecture in bias propagation during language modeling. |
| Outcome: | The proposed method analyzes the interaction between training data and model architecture in bias propagation during language modeling. |
Do Neural Language Models Overcome Reporting Bias? (2020.coling-main)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained language models can overcome reporting bias by estimating the plausibility of rare but unspoken facts. |
| Approach: | They revisit the experiments conducted by Gordon and Van Durme (2013) . they find that pre-trained language models overestimate the very rare . |
| Outcome: | The proposed approach overestimates the rare at the expense of the rare, while minimizing reporting bias. |
Societal Biases in Language Generation: Progress and Challenges (2021.acl-long)
Copied to clipboard
| Challenge: | Language generation techniques can produce undesirable societal biases that can negatively impact marginalized populations. |
| Approach: | They propose to examine how decoding techniques contribute to biases in language generation . they also conduct experiments to quantify the effects of these techniques . |
| Outcome: | The proposed methods can reduce biases and improve user experience, the authors argue . they also show that the proposed techniques can reduce societal biase . |
Adaptive Bridge between Training and Inference for Dialogue Generation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that our model can achieve a significant improvement in terms of metric-based evaluation and human evaluation compared with the state-of-the-art exposure bias approaches. |
| Approach: | They propose a novel adaptive switching mechanism which automatically transits between ground-truth learning and generated learning regarding the word-level matching score. |
| Outcome: | The proposed model improves on Chinese and English reddit datasets compared with state-of-the-art models on the word-level matching score. |
Addressing Bias and Hallucination in Large Language Models (2024.lrec-tutorials)
Copied to clipboard
Nihar Ranjan Sahoo, Ashita Saxena, Kishan Maharaj, Arif A. Ahmad, Abhijit Mishra, Pushpak Bhattacharyya
| Challenge: | This tutorial provides a comprehensive overview of two critical aspects of Large Language Models: bias and hallucination. |
| Approach: | This tutorial provides an overview of two critical aspects of Large Language Models: bias and hallucination. |
| Outcome: | This tutorial delves into the complex dimensions of Large Language Models (LLMs) it outlines ethical considerations pertinent to their development and discusses hallucination, a prevalent issue in generative AI systems such as LLMs. |
Cognitive Effects and Biases in Large Language Models (2026.eacl-tutorials)
Copied to clipboard
| Challenge: | This tutorial bridges psychology and NLP to clarify cognitive effects and biases in large language models. |
| Approach: | This tutorial bridges psychology and NLP to clarify cognitive effects and biases in large language models. |
| Outcome: | This tutorial bridges psychology and NLP to clarify cognitive effects and biases in large language models. |
The Impact of Inference Acceleration on Bias of LLMs (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent work suggests strategies to increase inference efficiency with LLMs . however, these strategies may inadvertently lead to some side-effects. |
| Approach: | They propose to optimize inference acceleration strategies such as quantization, pruning, and caching to reduce inference cost and latency while maintaining predictive performance. |
| Outcome: | The proposed strategies reduce cost and latency while maintaining predictive performance while preserving the model size. |
A Systematic Review of Reproducibility Research in Natural Language Processing (2021.eacl-main)
Copied to clipboard
| Challenge: | Despite the recent progress in reproducibility, the field is far from reaching a consensus on how reproducibility should be defined, measured and addressed. |
| Approach: | They propose to provide a wide-angle snapshot of current work on reproducibility in NLP. |
| Outcome: | The proposed work will provide a wide-angle snapshot of current work on reproducibility in NLP. |