No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytelling (P18-1)
Copied to clipboard
| Challenge: | Visual captioning is aimed at depicting the concrete content of images, but its capability of performing human-like understanding is still restrictive. |
| Approach: | They propose an Adversarial REward Learning framework to learn an implicit reward function from human demonstrations and optimize policy search with the learned reward function. |
| Outcome: | The proposed framework improves performance over state-of-the-art (SOTA) methods in cloning expert behaviors, but human evaluation shows that it achieves significant improvement in generating more human-like stories than SOTA systems. |
Similar Papers
Evaluating Rewards for Question Generation Models (N19-1)
Copied to clipboard
| Challenge: | Recent approaches to question generation have used modifications to a Seq2Seq architecture inspired by advances in machine translation. |
| Approach: | They propose to use a Seq2Seq architecture to train models to generate one-step-ahead predictions, but at test time, the model is asked to generate a whole sequence, causing errors to propagate through the generation process. |
| Outcome: | The proposed model is trained to generate a plausible question, conditioned on an input document and answer span within that document. |
Plot-guided Adversarial Example Construction for Evaluating Open-domain Story Generation (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to generate implausible stories using plots are unnatural and oversimplify the characteristics of implusible machine-generated stories. |
| Approach: | They propose to generate a more comprehensive set of implausible stories using plots . plots are structured representations of controllable factors used to generate stories . |
| Outcome: | The proposed model improves the quality of generated implausible stories using plots . it shows that the evaluation metrics trained on the generated data correlate better with human judgments compared to baselines. |
Inverse Reinforcement Learning for Text Summarization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies show that inverse reinforcement learning (RL) training has certain disadvantages such as object mismatch and exposure bias. |
| Approach: | They propose inverse reinforcement learning (IRL) as an effective paradigm for training abstractive summarization models. |
| Outcome: | The proposed model outperforms MLE and RL baselines on ROUGE, coverage, novelty, compression ratio, factuality, and human evaluations. |
Exploring Question-Specific Rewards for Generating Deep Questions (2020.coling-main)
Copied to clipboard
| Challenge: | Recent question generation approaches use the sequence-to-sequence framework to optimize the log likelihood of ground-truth questions using teacher forcing. |
| Approach: | They propose to optimize for QG-specific objectives via reinforcement learning to improve question quality. |
| Outcome: | The proposed model improves the fluency, relevance, and answerability of generated questions. |
RLHF Algorithms Ranked: An Extensive Evaluation Across Diverse Tasks, Rewards, and Hyperparameters (2025.emnlp-industry)
Copied to clipboard
Lucas Spangher, Rama Kumar Pasumarthi, Nick Masiewicki, William F. Arnold, Aditi Kaushal, Dale Johnson, Peter Grabowski, Eugene Ie
| Challenge: | Proximal Policy Optimization (PPO) has fallen out of favor for Large Language Models (LLMs), but its complexity and inefficiency have spurred the investigation of simpler alternatives. |
| Approach: | They evaluate 17 RLHF algorithms on two benchmarks, OpenAI’s TL;DR Summarization and Anthropic’s Helpfulness / Harmlessness. |
| Outcome: | The proposed methods are based on OpenAI’s TL;DR Summarization and Anthropic’s Helpfulness / Harmlessness benchmarks with two different reward models and a Rules based reward model. |
Rewarding Smatch: Transition-Based AMR Parsing with Reinforcement Learning (P19-1)
Copied to clipboard
| Challenge: | Abstract meaning representations (AMRs) are labeled directed acyclic graphs that represent a non intersentential abstraction of natural language with broad-coverage semantic representations. |
| Approach: | They build upon a transition-based AMR parser that uses Stack-LSTMs and augment training with policy learning. |
| Outcome: | The proposed parser performs comparable to the best published parsers. |
Reward Gaming in Conditional Text Generation (2023.acl-long)
Copied to clipboard
| Challenge: | Recent work has used reward functions learned from human annotations to align conditional text generation models with desired behaviors. |
| Approach: | They propose to use reinforcement learning to train conditional text generation models with reward functions learned from human annotations to align outputs with desired behaviors. |
| Outcome: | The proposed framework improves the quality of generated summaries by using saliency and faithfulness metrics. |
How Helpful is Inverse Reinforcement Learning for Table-to-Text Generation? (2021.acl-short)
Copied to clipboard
| Challenge: | Existing approaches to Table-to-Text generation suffer from issues such as missing information, repetition and repetition. |
| Approach: | They propose to use Inverse Reinforcement Learning (IRL) to solve the Table-to-Text task . they use multiple interpretable unsupervised reward components that are combined linearly to form a composite reward function. |
| Outcome: | The proposed task outperforms strong RL baselines marginally in the Table-to-Text task. |
ReinforceBug: A Framework to Generate Adversarial Textual Examples (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent studies have demonstrated that ML Models are sensitive to Adversarial Examples (AEs) AEs are generated by perturbingining examples that preserve the intrinsic utility of the ML solutions but influence the classifier's predictions between original and modified inputs. |
| Approach: | They propose a reinforcement learning framework that learns a policy that is transferable on unseen datasets and generates utility-preserving and transferable AEs. |
| Outcome: | The proposed framework is 10% more successful than the state-of-the-art attack TextFooler. |
Rewarding Semantic Similarity under Optimized Alignments for AMR-to-Text Generation (2022.acl-short)
Copied to clipboard
| Challenge: | Automatic evaluation metrics score natural language generation systems based on how well they lexically align to humanannotated references. |
| Approach: | They propose to replace greedy alignments in BERTScore with optimized ones that replace the n-gram matching BERTAcore metrics with a token embedding to prevent domain mismatch. |
| Outcome: | The proposed metrics outperform cross-entropy and BLEU reward baselines on AMR-to-text generation. |