Local Temperature Beam Search: Avoid Neural Text DeGeneration via Enhanced Calibration (2023.findings-acl)
Copied to clipboard
Dongkyu Lee, Gyeonghun Kim, Janghoon Han, Taesuk Hong, Yi-Reun Kim, Stanley Jungkyu Choi, Nevin L. Zhang
| Challenge: | Existing approaches to inference have been based on stochastic decoding but they sacrifice output quality due to randomness. |
| Approach: | They propose a deterministic decoding scheme, local temperature beam search, which reduces repetition while maintaining the level of coherence as in beam search. |
| Outcome: | The proposed inference scheme reduces repetition while maintaining coherence as in beam search. |
Similar Papers
Incremental Beam Manipulation for Natural Language Generation (2021.eacl-main)
Copied to clipboard
| Challenge: | a larger beam size can lead to deteriorating performance of natural language generation systems due to model errors . performance of NLG systems can plateau or even decrease when beam sizes larger than 10 are used . |
| Approach: | They propose to rerank the output of beam search to produce a good set of hypotheses . they propose incremental beam manipulation to discarded hypothese . |
| Outcome: | The proposed method outperforms a strong reranker on the E2E and WebNLG datasets while being on par with the existing method. |
If beam search is the answer, what was the question? (2020.emnlp-main)
Copied to clipboard
| Challenge: | surprisingly, beam search results on language generation tasks are low-quality . despite its high error rate, beam searches can be used to decode models with high probability . |
| Approach: | They frame beam search as the exact solution to a different decoding objective . they propose a set of decoding objectives that explicitly enforce this property . |
| Outcome: | The proposed method enforces uniform information density in text, a property motivated by cognitive science. |
On Decoding Strategies for Neural Text Generators (2022.tacl-1)
Copied to clipboard
| Challenge: | a recent study suggests that decoding strategies may be more important than the model architecture itself when generating text from probabilistic models. |
| Approach: | They propose to measure changes in attributes of generated text as a function of decoding strategy and task using human and automatic evaluation. |
| Outcome: | The proposed study shows that decoding strategies do not always transfer across tasks . authors show that the differences in attributes are not always consistent across tasks, they say . |
Machine Translation Decoding beyond Beam Search (2021.emnlp-main)
Copied to clipboard
Rémi Leblond, Jean-Baptiste Alayrac, Laurent Sifre, Miruna Pislar, Lespiau Jean-Baptiste, Ioannis Antonoglou, Karen Simonyan, Oriol Vinyals
| Challenge: | a new study examines whether beam search can be replaced by a more powerful metric-driven search technique. |
| Approach: | They propose a beam search method which is agnostic to the end metric and report results on a variety of metrics. |
| Outcome: | The proposed method is based on a Monte-Carlo Tree Search (MCTS) based method and shows it can be used in language applications. |
Improving Beam Search by Removing Monotonic Constraint for Neural Machine Translation (P18-2)
Copied to clipboard
| Challenge: | a beam search algorithm produces monotonic left-to-right order, meaning a hypothesis cannot be revisited . a proposed algorithm allows discarded hypotheses to be recovered in a later step. |
| Approach: | They propose to decode a beam search algorithm that considers multiple hypotheses simultaneously . they propose to maintain all found hypothese a single priority queue and a universal score function . |
| Outcome: | The proposed algorithm improves translations even for high-performance models in English-Japanese translation task. |
A Call for Clarity in Beam Search: How It Works and When It Stops (2024.lrec-main)
Copied to clipboard
| Challenge: | Empirical results show that a modified beam decoding implementation improves decoding performance of strong, neural language generation models. |
| Approach: | They propose a modification to a beam decoding implementation that generalizes the stopping criterion and provides flexibility to the depth of search. |
| Outcome: | The proposed method improves decoding performance of strong models on news text summarization and machine translation over diverse language pairs with negligible inference slowdown. |
Consistency of a Recurrent Language Model With Respect to Incomplete Decoding (2020.emnlp-main)
Copied to clipboard
| Challenge: | Neural sequence models trained with maximum likelihood have been shown to exhibit issues such as length bias and degenerate repetition. |
| Approach: | They propose to use a recurrent language model to address inconsistency in decoding algorithms that are inconsistent despite the fact that recursive language models are trained to produce sequences of finite length. |
| Outcome: | The proposed methods prevent inconsistency in the proposed models. |
Exploring Recombination for Efficient Decoding of Neural Machine Translation (D18-1)
Copied to clipboard
| Challenge: | Neural Machine Translation (NMT) decoder captures features of entire prediction history . some partial hypotheses with different prefixes will be regarded differently no matter how similar they are . |
| Approach: | They propose a method that uses a n-gram suffix to adapt it to beam search decoding. |
| Outcome: | The proposed method can obtain similar translation quality with a smaller beam size, making it more efficient. |
Characterizing and addressing the issue of oversmoothing in neural autoregressive sequence modeling (2022.aacl-main)
Copied to clipboard
| Challenge: | Neural autoregressive sequence models assign high probability to unreasonably short sequences . authors propose to minimize oversmoothing rate during training . |
| Approach: | They propose to minimize the oversmoothing rate during training by tuning the regularization strength. |
| Outcome: | The proposed regularization can control the oversmoothing rate and improve decoding performance. |
Adaptive Contrastive Search: Uncertainty-Guided Decoding for Open-Ended Text Generation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to decode text to the most probable sequence have been proposed to address these challenges by improving coherence, diversity, and resemblance to human-generated text. |
| Approach: | They propose a novel decoding strategy that extends contrastive search by incorporating an adaptive degeneration penalty informed by the model’s estimated uncertainty at each generation step. |
| Outcome: | The proposed approach improves creativity and coherence while maintaining coherency across model architectures, languages, and datasets. |