| Challenge: | Conditional language models can generate a diverse set of outputs, but for open-ended tasks, beam search is ill-suited to generating a set of diverse sequences. |
| Approach: | They propose a method where we over-sample candidates and use clustering to remove similar sequences to achieve high diversity without sacrificing quality. |
| Outcome: | The proposed method over-samples candidates and removes similar sequences to achieve high diversity without sacrificing quality. |
Similar Papers
Massive-scale Decoding for Text Generation using Lattices (2022.naacl-main)
Copied to clipboard
| Challenge: | Conditional neural text generation models generate high-quality outputs, but often focus on a mode when what we really want is a diverse set of options. |
| Approach: | They propose a search algorithm to construct lattices encoding a massive number of generation options. |
| Outcome: | The proposed algorithm encodes thousands of diverse options that remain grammatical and high-quality into one lattice. |
On Decoding Strategies for Neural Text Generators (2022.tacl-1)
Copied to clipboard
| Challenge: | a recent study suggests that decoding strategies may be more important than the model architecture itself when generating text from probabilistic models. |
| Approach: | They propose to measure changes in attributes of generated text as a function of decoding strategy and task using human and automatic evaluation. |
| Outcome: | The proposed study shows that decoding strategies do not always transfer across tasks . authors show that the differences in attributes are not always consistent across tasks, they say . |
Twist Decoding: Diverse Generators Guide Each Other (2022.emnlp-main)
Copied to clipboard
Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Hao Peng, Ximing Lu, Dragomir Radev, Yejin Choi, Noah A. Smith
| Challenge: | Using a variety of language generation models, ensembling models is challenging during inference. |
| Approach: | They propose a method that decodes text models that do not assume a shared vocabulary, tokenization or generation order. |
| Outcome: | The proposed method outperforms models decoded in isolation over various scenarios. |
A Well-Composed Text is Half Done! Composition Sampling for Diverse Conditional Generation (2022.acl-long)
Copied to clipboard
Shashi Narayan, Gonçalo Simões, Yao Zhao, Joshua Maynez, Dipanjan Das, Michael Collins, Mirella Lapata
| Challenge: | Composition Sampling is a simple but effective method to generate diverse outputs for conditional generation of higher quality compared to previous stochastic decoding strategies. |
| Approach: | They propose a method to generate diverse outputs for conditional generation . they use a plan-based neural generation model that is trained to create a composition of the output and then generate by conditioning on it and the input. |
| Outcome: | The proposed method avoids text degeneration by first sampling a composition in the form of an entity chain and then using beam search to generate the best possible text grounded to this entity chain. |
How Diversely Can Language Models Solve Problems? Exploring the Algorithmic Diversity of Model-Generated Code (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Language models (LMs) have exhibited impressive abilities in generating code from natural language requirements. |
| Approach: | They propose to introduce various metrics with inter-code similarity to evaluate the diversity of generated code by comparing model-generated solutions with human-written ones. |
| Outcome: | The proposed method leverages LMs’ capabilities in code understanding and reasoning, resulting in a set of metrics that represent the number of algorithms in model-generated solutions. |
Mixup Decoding for Diverse Machine Translation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for generating multiple translations for source and target languages neglect the one-to-many mapping between the source and the target languages. |
| Approach: | They propose a method to generate different translations for the input sentence by linearly interpolating it with different sentence pairs sampled from the training corpus during decoding. |
| Outcome: | Experiments on WMT’16 en-ro, WMT'14 en de, and WMT ‘17 zh-en show that the proposed method outperforms all previous diverse machine translation methods. |
First the Worst: Finding Better Gender Translations During Beam Search (2022.findings-acl)
Copied to clipboard
| Challenge: | Neural language generation models optimized by likelihood tend towards 'safe' word choice. |
| Approach: | They propose to use beam search to improve gender diversity in n-best lists and rerank n best lists using gender features obtained from the source sentence to address this problem. |
| Outcome: | The proposed approach improves gender diversity in n-best lists and reranks n best lists using gender features obtained from the source sentence. |
Decoding Matters: Addressing Amplification Bias and Homogeneity Issue in Recommendations for Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to adapt Large Language Models (LLMs) for recommendation encounter significant challenges such as amplification bias and homogeneity. |
| Approach: | They propose a new decoding approach called Debiasing-Diversifying Decoding (D3) that disables length normalization for ghost tokens to alleviate amplification bias and incorporates a text-free assistant model to encourage tokens less frequently generated by LLMs for counteracting recommendation homogeneity. |
| Outcome: | Extensive experiments on real-world datasets demonstrate the proposed approach’s effectiveness in enhancing accuracy and diversity. |
Decoding Decoded: Understanding Hyperparameter Effects in Open-Ended Text Generation (2025.coling-main)
Copied to clipboard
| Challenge: | Generative large language models generate a high-dimensional probability distribution over all tokens in their vocabulary. |
| Approach: | They conduct extensive sensitivity analyses to determine how hyperparameter choices shape the outputs of generative large language models. |
| Outcome: | The proposed methods influence the distribution of diversity and coherence metrics in human-written text, but the optimal configurations vary across models and tasks. |
GenTranslate: Large Language Models are Generative Multilingual Speech and Machine Translators (2024.acl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have stepped forward the development of multilingual speech and machine translation by its reduced representation errors and incorporated external knowledge. |
| Approach: | They propose a generative paradigm for translation tasks that integrates the diverse translation versions in N-best list. |
| Outcome: | The proposed model outperforms the state-of-the-art model on speech and machine translation benchmarks on various languages. |