Feedback Attribution for Counterfactual Bandit Learning in Multi-Domain Spoken Language Understanding (2021.emnlp-main)
Copied to clipboard
| Challenge: | a large amount of labeled data is needed for fine-tuning. |
| Approach: | They propose attribution methods inspired by multi-agent reinforcement learning for a feedback attribution problem in spoken language understanding. |
| Outcome: | The proposed methods can train competitive models from user feedback. |
Similar Papers
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback (P18-1)
Copied to clipboard
| Challenge: | a recent study shows that counterfactual learning from human bandit feedback can improve neural semantic parsers . cost and difficulty of manually preparing large amounts of parses is a bottleneck for supervised learning . |
| Approach: | They propose to use human bandit feedback to apply counterfactual learning to neural parsing . they devise an easy-to-use interface to collect human feedback on semantic parses . |
| Outcome: | The proposed framework improves semantic parsers by reducing the cost of manual parsing . the proposed framework is based on human bandit feedback collected by the user . |
Simulating Bandit Learning from User Feedback for Extractive Question Answering (2022.acl-long)
Copied to clipboard
| Challenge: | Explicit feedback from users can be used to continually improve system performance. |
| Approach: | They study the potential of learning from user feedback for extractive question answering by simulating feedback using supervised data. |
| Outcome: | The proposed model improves on a few examples and can be deployed in new domains without any data annotation effort. |
Context Attribution with Multi-Armed Bandit Optimization (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to augmenting attribution with retrieval-augmented generation (RAG) focus on training models to explicitly cite context segments during generation, but their reliability remains unverifiable. |
| Approach: | They propose a framework that formulates context attribution as a combinatorial multi-armed bandit problem by using Linear Thompson Sampling to efficiently identify the most influential context segments while minimizing the number of model queries. |
| Outcome: | The proposed method reduces model queries by 30% while matching or exceeding the attribution quality of existing approaches. |
BanditSum: Extractive Summarization as a Contextual Bandit (D18-1)
Copied to clipboard
| Challenge: | Existing methods for extractive summarization are heuristically generated and require a set of binary labels to be selected. |
| Approach: | They propose a method for training neural networks to perform single-document extractive summarization without heuristically-generated extractive labels. |
| Outcome: | The proposed method achieves better ROUGE scores than the state-of-the-art methods and significantly fewer update steps than competing approaches. |
Counterfactual Off-Policy Training for Neural Dialogue Generation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for open-domain dialogue generation suffer from data insufficiency . a potential response inferred in hindsight is called a counterfactual reasoning . |
| Approach: | They propose to explore potential responses by counterfactual reasoning . given an observed response, the model automatically infers the outcome of an alternative policy that could have been taken . |
| Outcome: | The proposed model outperforms the HRED model and conventional learning frameworks on the DailyDialog dataset. |
Navigating Noisy Feedback: Enhancing Reinforcement Learning with Error-Prone Language Models (2024.findings-emnlp)
Copied to clipboard
Muhan Lin, Shuyang Shi, Yue Guo, Behdad Chalaki, Vaishnav Tadiparthi, Ehsan Moradi Pari, Simon Stepputtis, Joseph Campbell, Katia Sycara
| Challenge: | Reward hacking is a problem in reinforcement learning where the ability to specify the desired behavior of a reward function is difficult. |
| Approach: | They propose to use feedback as a potential-based shaping function to solicit and apply feedback from large language models to improve convergence speed and policy returns. |
| Outcome: | The proposed method improves convergence speed and policy returns over baselines even with significant ranking errors and eliminates the need for complex post-processing of reward functions. |
Evaluating and Modeling Attribution for Cross-Lingual Question Answering (2023.emnlp-main)
Copied to clipboard
Benjamin Muller, John Wieting, Jonathan Clark, Tom Kwiatkowski, Sebastian Ruder, Livio Soares, Roee Aharoni, Jonathan Herzig, Xinyi Wang
| Challenge: | Open-retrieval question answering systems are lacking in attribution for cross-lingual question answering . open-research questions are available in 20 languages, but their raw generation often falls short in factuality . |
| Approach: | They are the first to study attribution for cross-lingual question answering . they collect data in 5 languages to assess the attribution level of a state-of-the-art QA system . |
| Outcome: | The proposed approach improves the attribution level of a state-of-the-art cross-lingual QA system. |
Connecting Attributions and QA Model Behavior on Realistic Counterfactuals (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent research in interpretability of neural models has yielded numerous token attribution techniques, but it is hard to evaluate whether these explanations are faithful. |
| Approach: | They propose to use pairwise attributions to connect outputs to high-level model behavior to examine how well different attribution techniques align with this assumption on realistic counterfactuals in the case of reading comprehension (RC). |
| Outcome: | The proposed methods are better suited to RC than token-level attributions across different RC settings, and the best performance comes from a modification that was proposed to an existing pairwise attribution method. |
Jump Starting Bandits with LLM-Generated Prior Knowledge (2024.emnlp-main)
Copied to clipboard
| Challenge: | Contextual multi-armed bandits generate personalized recommendations based on user-specific contexts. |
| Approach: | They propose an initialization algorithm for contextual bandits by prompting LLMs to produce a pre-training dataset of approximate human preferences for the bandit. |
| Outcome: | The proposed approach significantly reduces online learning regret and data-gathering costs for training such models. |
Making Long-Context Language Models Better Multi-Hop Reasoners (2024.acl-long)
Copied to clipboard
| Challenge: | Recent advances in long-context modeling have enhanced language models for complex tasks, but they struggle with multi-hop reasoning and noisy contexts. |
| Approach: | They propose an approach that prompts LMs to supply attributions for each assertion during reasoning. |
| Outcome: | The proposed model achieves competitive performance on multi-hop reasoning benchmarks, closely paralleling proprietary LMs such as ChatGPT and Claude-instant. |