Yue Dong, Chandra Bhagavatula, Ximing Lu, Jena D. Hwang, Antoine Bosselut, Jackie Chi Kit Cheung, Yejin Choi
| Challenge: | Degeneration of neural text is associated with insufficient learning of task-specific characteristics by the attention mechanism. |
| Approach: | They propose to use attention modulation to inject priors into inference to improve fluency, creativity, and commonsense reasoning in neural text generation models. |
| Outcome: | The proposed method improves fluency, creativity, and commonsense reasoning, and significantly reduces sentence-level repetition. |
Similar Papers
On Decoding Strategies for Neural Text Generators (2022.tacl-1)
Copied to clipboard
| Challenge: | a recent study suggests that decoding strategies may be more important than the model architecture itself when generating text from probabilistic models. |
| Approach: | They propose to measure changes in attributes of generated text as a function of decoding strategy and task using human and automatic evaluation. |
| Outcome: | The proposed study shows that decoding strategies do not always transfer across tasks . authors show that the differences in attributes are not always consistent across tasks, they say . |
Coherence boosting: When your pretrained language model is not paying enough attention (2022.acl-long)
Copied to clipboard
| Challenge: | Long-range semantic coherence remains a challenge in automatic language generation and understanding. |
| Approach: | They propose a procedure that increases a model’s focus on a long context by distributional analyses of generated ordinary text and dialog responses. |
| Outcome: | The proposed procedure increases the model's focus on a long context. |
Through the Looking Glass: Learning to Attribute Synthetic Text Generated by Language Models (2021.eacl-main)
Copied to clipboard
| Challenge: | Recent advances in natural language processing have enabled synthetic text generation that is often comparable to the organic text. |
| Approach: | They propose and test several ML-based methods to attribute authorship of synthetic text to language models (LMs) they propose to use a fine-tuned version of XLNet to achieve excellent accuracy . |
| Outcome: | The proposed method achieves excellent accuracy (91% to near perfect 98%) across a range of experiments where the synthetic text may be generated using pre-trained LMs, fine-tuned LM, or by varying text generation parameters. |
The Amazing World of Neural Language Generation (2020.emnlp-tutorials)
Copied to clipboard
| Challenge: | Recent years have seen a paradigm shift in neural text generation due to advances in deep contextual language modeling and transfer learning. |
| Approach: | They will discuss how and why NLG models succeed/fail at generating coherent text. |
| Outcome: | This paper will discuss how and why these models succeed/fail at generating coherent text, and provide insights on several applications. |
Towards Content Transfer through Grounded Text Generation (N19-1)
Copied to clipboard
| Challenge: | Recent work in neural natural language generation has attracted significant interest in controlling the form of text, such as style, persona, and wordiness. |
| Approach: | They propose a task where the task is to generate a next sentence in a document that fits its context and is grounded in . external textual source such as a news story. |
| Outcome: | The proposed task is based on 640k Wikipedia referenced sentences paired with the source articles to show significant improvements against baselines. |
Attention Entropy is a Key Factor: An Analysis of Parallel Context Encoding with Full-attention-based Pre-trained Language Models (2025.acl-long)
Copied to clipboard
Zhisong Zhang, Yan Wang, Xinting Huang, Tianqing Fang, Hongming Zhang, Chenlong Deng, Shuaiyi Li, Dong Yu
| Challenge: | Large language models have demonstrated remarkable performance across a wide range of language tasks due to their remarkable ability in context modeling. |
| Approach: | They propose to use parallel context encoding to reduce attention entropy by incorporating attention sinks and selective mechanisms to reduce irregular attention . they also propose to incorporate attention sink mechanisms into the parallel encoded context to reduce the irregular attention. |
| Outcome: | The proposed methods lower irregular attention entropy and narrow performance gaps. |
Hierarchical Neural Story Generation (P18-1)
Copied to clipboard
| Challenge: | a hierarchical model that generates a premise and then conditions on it creates fluent text . a novel form of model fusion improves the relevance of the story to the prompt . |
| Approach: | They use a hierarchical model that first generates a premise, then transforms it into a text . they use fusion to improve relevance of the story to the prompt and add a gated mechanism to model context . |
| Outcome: | The proposed model improves on strong baselines on automated and human evaluations. |
Neural Data-to-Text Generation via Jointly Learning the Segmentation and Correspondence (2020.acl-main)
Copied to clipboard
| Challenge: | Recent neural attention models conflate all steps into a single end-to-end system and simplify training process. |
| Approach: | They propose to explicitly segment target text into fragment units and align them with their data correspondences. |
| Outcome: | The proposed model outperforms neural attention models on E2E and WebNLG benchmarks. |
Sentence-Level Content Planning and Style Specification for Neural Text Generation (D19-1)
Copied to clipboard
| Challenge: | Recent advances in text generation systems often produce incoherent and unfaithful outputs . a novel automated text generation system takes into account content selection, text planning, and surface realization. |
| Approach: | They propose an end-to-end trained two-step text generation model that considers sentence-level content planners and language styles. |
| Outcome: | The proposed model outperforms competing models in three domains with diverse topics and varying language styles. |
Knowledge-Enriched Natural Language Generation (2021.emnlp-tutorials)
Copied to clipboard
| Challenge: | Knowledge-enriched text generation poses unique challenges in modeling and learning . a roadmap will outline the state-of-the-art methods to tackle these challenges . |
| Approach: | They propose a roadmap to tackle the challenges of knowledge-enriched text generation . they will dive deep into various technical components to illustrate how to represent knowledge . |
| Outcome: | This tutorial outlines the state-of-the-art methods to tackle the problem . it aims to show how to represent knowledge, feed knowledge into a generation model, evaluate results . |