| Challenge: | Existing methods for conditional text generation suffer from large action space and delayed reward, as the reward can be computed only after an entire sequence is generated. |
| Approach: | They propose a method that provides partial rewards for intermediate actions taken on partial sequences to prioritize actions that lead to the generation of more desirable sequences. |
| Outcome: | The proposed method overcomes the limitations of the prevalent supervised maximum likelihood estimation approach. |
Similar Papers
Reward Gaming in Conditional Text Generation (2023.acl-long)
Copied to clipboard
| Challenge: | Recent work has used reward functions learned from human annotations to align conditional text generation models with desired behaviors. |
| Approach: | They propose to use reinforcement learning to train conditional text generation models with reward functions learned from human annotations to align outputs with desired behaviors. |
| Outcome: | The proposed framework improves the quality of generated summaries by using saliency and faithfulness metrics. |
Learning to Selectively Learn for Weakly Supervised Paraphrase Generation with Model-based Reinforcement Learning (2022.naacl-main)
Copied to clipboard
| Challenge: | Paraphrase generation is an important natural language generation task . however, the effectiveness of paraphrase generation can be limited due to the limited data available. |
| Approach: | They propose a weakly supervised approach to paraphrase generation that leverages reinforcement learning for effective model training with data selection. |
| Outcome: | The proposed model improves the state-of-the-art performance on four weakly supervised paraphrase generation tasks. |
How Helpful is Inverse Reinforcement Learning for Table-to-Text Generation? (2021.acl-short)
Copied to clipboard
| Challenge: | Existing approaches to Table-to-Text generation suffer from issues such as missing information, repetition and repetition. |
| Approach: | They propose to use Inverse Reinforcement Learning (IRL) to solve the Table-to-Text task . they use multiple interpretable unsupervised reward components that are combined linearly to form a composite reward function. |
| Outcome: | The proposed task outperforms strong RL baselines marginally in the Table-to-Text task. |
Text Simplification with Reinforcement Learning Using Supervised Rewards on Grammaticality, Meaning Preservation, and Simplicity (2020.aacl-srw)
Copied to clipboard
| Challenge: | Existing studies in text-to-text generation do not align with human-perspectives for these perspectives. |
| Approach: | They propose to use BERT regressors fine-tuned for grammaticality, meaning preservation, and simplicity as reward estimators to optimize rewards for text simplification. |
| Outcome: | The proposed method achieves text simplification conforming to human-perspectives. |
Enhancing Generative Retrieval with Reinforcement Learning from Relevance Feedback (2023.emnlp-main)
Copied to clipboard
| Challenge: | End-to-end generative retrieval models produce document identifiers in response to a query . however, this approach has two challenges: an overemphasis on top-1 results at the expense of overall ranking quality. |
| Approach: | They propose a generative retrieval model with reinforcement learning from relevance feedback to align token-level docid generation with document-level relevance estimation. |
| Outcome: | The proposed model aligns token-level docid generation with document-level relevance estimation. |
Generative Pretraining for Paraphrase Evaluation (2022.acl-long)
Copied to clipboard
| Challenge: | ParaBLEU is a paraphrase representation learning model and evaluation metric for text generation. |
| Approach: | They propose a paraphrase representation learning model and evaluation metric for text generation that uses generative conditioning as a pretraining objective. |
| Outcome: | The proposed model outperforms existing models on the 2017 WMT Metrics Shared Task using only 50% of the available training data and surpasses BLEU, ROUGE and METEOR with only 40 examples. |
Reinforcement Learning with Token-level Feedback for Controllable Text Generation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for controllable text generation are guided by coarse-grained feedback, which may lead to suboptimal performance owing to semantic twists or progressions within sentences. |
| Approach: | They propose a reinforcement learning algorithm which formulates TOken-LEvel rewards for controllable text generation and employs a "first-quantize-then-noise" paradigm to enhance the robustness of the RL algorithm. |
| Outcome: | The proposed algorithm can achieve superior performance on single-attribute and multi-attract control tasks. |
Learning from Bootstrapping and Stepwise Reinforcement Reward: A Semi-Supervised Framework for Text Style Transfer (2022.findings-naacl)
Copied to clipboard
| Challenge: | Text style transfer is an important task in controllable language generation due to the scarcity of large-scale parallel data. |
| Approach: | They propose a semi-supervised framework for text style transfer that bootstraps with supervision guided by automatically constructed pseudo-parallel pairs and improves the sequence-to-sequence policy gradient via reinforcement rewards. |
| Outcome: | The proposed framework achieves state-of-the-art performance on multiple datasets and produces effective generation with as minimal as 10% of training data. |
Learning and Analyzing Generation Order for Undirected Sequence Models (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Undirected neural sequence models generate monotonically from left to right in machine translation tasks. |
| Approach: | They train a policy that learns the generation order for a pre-trained, undirected translation model via reinforcement learning. |
| Outcome: | The proposed policy outperforms heuristic generation orders on three out of four language pairs. |
Joint Generator-Ranker Learning for Natural Language Generation (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for text generation train the generator and ranker individually . existing methods neglect the mutual feedback that could enhance the quality of outputs . |
| Approach: | They propose a joint training algorithm that integrates the generator and ranker in a single framework. |
| Outcome: | The proposed algorithm surpasses existing methods on four public datasets across three common generation scenarios. |