A Frustratingly Simple Decoding Method for Neural Text Generation (2024.lrec-main)
Copied to clipboard
| Challenge: | Neural text generation is notorious for repetitive loops and tedious outputs. |
| Approach: | They propose a method that penalizes future generation of repetitive content . they construct an anti-LM based on previously generated text . |
| Outcome: | The proposed method outperforms established baselines in terms of generation quality, decoding speed, and universality. |
Similar Papers
Critic-Driven Decoding for Mitigating Hallucinations in Data-to-text Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Hallucination of text lacking grounding in input data is a problem in neural data-to-text generation. |
| Approach: | They propose to combine probabilistic output of a generator language model with the output of an “text critic” classifier which guides the generation by assessing the match between the input data and the generated text. |
| Outcome: | The proposed method improves on the WebNLG and OpenDialKG benchmarks. |
A Simple Recipe towards Reducing Hallucination in Neural Surface Realisation (P19-1)
Copied to clipboard
| Challenge: | Recent neural language generation systems often hallucinate contents when trained on loosely corresponding pairs of the input structure and text. |
| Approach: | They propose to integrate a language understanding module for data refinement with self-training iterations to induce strong equivalence between the input data and the paired text. |
| Outcome: | Experiments on the E2E challenge dataset show that the proposed framework reduces relative unaligned noise by 50% compared with the current state-of-the-art ensemble generator. |
SSSD: Simply-Scalable Speculative Decoding (2026.acl-long)
Copied to clipboard
Michele Marzollo, Jiawei Zhuang, Niklas Roemer, Niklas Zwingenberger, Lorenz K Muller, Lukas Cavigelli
| Challenge: | Existing methods for accelerating inference in Large Language Models require additional training and training, resulting in a higher deployment and maintenance cost. |
| Approach: | They propose a training-free method that combines lightweight n-gram matching with hardware-aware speculation. |
| Outcome: | SSSD reduces latency by up to 2.9 and is faster than autoregressive decoding methods. |
NEUROSTRUCTURAL DECODING: Neural Text Generation with Structural Constraints (2023.acl-long)
Copied to clipboard
| Challenge: | Current approaches for conditional text generation focus on lexical constraints, but lack syntactic constraints to support complex semantic constraints. |
| Approach: | They propose a decoding algorithm that incorporates syntactic constraints to improve the quality of the generated text. |
| Outcome: | The proposed method improves on three different language generation tasks and shows improved lexical and syntactic metrics. |
NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraints (2021.naacl-main)
Copied to clipboard
| Challenge: | Conditional text generation often requires lexical constraints, i.e., which words should or shouldn't be included in the output text. |
| Approach: | They propose an algorithm that enables neural language models to generate fluent text while satisfying complex lexical constraints. |
| Outcome: | The proposed algorithm outperforms existing methods on four benchmarks and shows that it handles any set of lexical constraints expressible under predicate logic while its asymptotic runtime is equivalent to conventional beam search. |
Contrastive Decoding: Open-ended Text Generation as Optimization (2023.acl-long)
Copied to clipboard
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, Mike Lewis
| Challenge: | Using a language model, maximum probability is a poor decoding objective because it produces short and repetitive text. |
| Approach: | They propose a reliable decoding approach that optimizes a contrastive objective subject to a plausibility constraint. |
| Outcome: | The proposed approach outperforms four strong decoding algorithms in automatic and human evaluations across wikipedia, news and story domains. |
Unlocking Anticipatory Text Generation: A Constrained Approach for Large Language Models Decoding (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models have shown a powerful ability for text generation, but undesired behaviors such as toxicity and hallucinations can manifest. |
| Approach: | They propose to formalize text generation as a future-constrained generation problem to minimize undesirable behaviors and enforce faithfulness to instructions. |
| Outcome: | The proposed approach is effective across three tasks, including keyword-constrained generation, toxicity reduction, and factual correctness in question-answering. |
Controlled Hallucinations: Learning to Generate Faithfully from Noisy Data (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Neural text generation (data- or text-to-text) demonstrates remarkable performance when training data is abundant which for many applications is not the case. |
| Approach: | They propose a technique to treat hallucinations as a controllable aspect of the generated text without dismissing any input and without modifying the model architecture. |
| Outcome: | The proposed technique can be used on a WikiBio dataset and in a human evaluation. |
Implicit Unlikelihood Training: Improving Neural Text Generation with Reinforcement Learning (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing approaches to language modeling use autoregressive methods, but they can produce repetitive results. |
| Approach: | They propose to add a loss function for regularization to avoid unwanted properties, such as contradiction or repetition, to a language model by using policy gradient reinforcement learning. |
| Outcome: | The proposed method reduces repetition without impacting the language model quality. |
Simple and Effective Noisy Channel Modeling for Neural Machine Translation (D19-1)
Copied to clipboard
| Challenge: | Previous work on noisy channel modeling relied on latent variable models that incrementally process the source and target sentence. |
| Approach: | They propose to use a standard sequence to sequence model which utilizes the entire source and target sentences to estimate posterior probability of a target sequence y given a source sequence x. |
| Outcome: | The proposed model outperforms direct models on German-English translations by up to 3.2 BLEU on four language pairs. |