| Challenge: | Existing approaches to decode open-ended text have addressed degeneration problems in large-scale language models (LLMs) |
| Approach: | They propose an improved decoding algorithm that leverages the Kullback–Leibler divergence to track the distribution distance between current and historical decoding steps. |
| Outcome: | The proposed algorithm outperforms existing methods in document continuation and story generation. |
Similar Papers
Contrastive Decoding: Open-ended Text Generation as Optimization (2023.acl-long)
Copied to clipboard
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, Mike Lewis
| Challenge: | Using a language model, maximum probability is a poor decoding objective because it produces short and repetitive text. |
| Approach: | They propose a reliable decoding approach that optimizes a contrastive objective subject to a plausibility constraint. |
| Outcome: | The proposed approach outperforms four strong decoding algorithms in automatic and human evaluations across wikipedia, news and story domains. |
Event Transition Planning for Open-ended Text Generation (2022.findings-acl)
Copied to clipboard
| Challenge: | Open-ended text generation tasks require models to generate coherent continuation given limited preceding context. |
| Approach: | They propose a novel two-stage method which explicitly arranges ensuing events in open-ended text generation tasks. |
| Outcome: | The proposed method improves coherence and diversity of open-ended text generation tasks. |
Mitigating the Learning Bias towards Repetition by Self-Contrastive Training for Open-Ended Generation (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing language models generate repetitive texts with greedy decoding or beam search. |
| Approach: | They propose a self-contrastive training technique to penalize the output of a premature checkpoint of the same model when it incorrectly predicts repetition. |
| Outcome: | The proposed training mitigates repetition while maintaining fluency while minimizing the overestimation of token-level repetition probabilities. |
Penalty Decoding: Well Suppress the Self-Reinforcement Effect in Open-Ended Text Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results demonstrate the efficacy of our approach in generating high-quality sentences resembling human output. |
| Approach: | They propose a forgetting mechanism that disregards distant tokens, reducing the burden of penalty selection. |
| Outcome: | The proposed approach generates high-quality sentences resembling human output. |
Adaptive Contrastive Search: Uncertainty-Guided Decoding for Open-Ended Text Generation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to decode text to the most probable sequence have been proposed to address these challenges by improving coherence, diversity, and resemblance to human-generated text. |
| Approach: | They propose a novel decoding strategy that extends contrastive search by incorporating an adaptive degeneration penalty informed by the model’s estimated uncertainty at each generation step. |
| Outcome: | The proposed approach improves creativity and coherence while maintaining coherency across model architectures, languages, and datasets. |
NeuralREG: An end-to-end approach to referring expression generation (P18-1)
Copied to clipboard
| Challenge: | Referring Expression Generation models typically rely on features such as salience and grammatical function to make decisions about form and content. |
| Approach: | They propose a new approach that makes decisions about form and content in one go . they use a delexicalized version of the WebNLG corpus to test the approach . |
| Outcome: | The proposed approach significantly improves over two strong baselines. |
Have Your Text and Use It Too! End-to-End Neural Data-to-Text Generation with Semantic Fidelity (2020.coling-main)
Copied to clipboard
| Challenge: | End-to-end neural data-totext generation has faced challenges generalizing to new domains and generating semantically consistent text. |
| Approach: | They propose a neural data-to-text generation system that makes minimal assumptions about the data representation and target domain. |
| Outcome: | The proposed system achieves state of the art results on four major D2T datasets with better semantic fidelity than the state-of-the-art methods. |
An Answer is just the Start: Related Insight Generation for Open-Ended Document-Grounded QA (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing QA benchmarks do not explicitly support document-grounded related insight generation . Existing document-based QA efforts focus on answering fact-based questions . |
| Approach: | They propose a task to generate additional insights from a document collection that improves, extends or rethinks an initial answer to an open-ended question. |
| Outcome: | The proposed task improves, extends, or rethinks an answer to an open-ended question. |
kNN-LM Does Not Improve Open-ended Text Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Interpolation-based retrieval-augmented language models (LMs) are a subtype of retrieval augmented language model that computes the probability of the next token by interpolating between the softmax distribution of the original LM and a token distribution formed by retrieving over an external datastore. |
| Approach: | They propose to interpolate the predicted distribution of the next word with a distribution formed from the most relevant retrievals for a given prefix. |
| Outcome: | The proposed methods do not exhibit improvements in open-ended generation quality, as measured by automatic evaluation metrics and human evaluations. |
Online Back-Parsing for AMR-to-Text Generation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Abstract meaning representation (AMR) is a semantic graph representation that abstracts meaning away from a sentence. |
| Approach: | They propose a decoder that back predicts projected AMR graphs on target sentences . their results show superiority over previous state-of-the-art decoded graph Transformer . |
| Outcome: | The proposed model outperforms the state-of-the-art model on two AMR benchmarks. |