Papers by Florian Strub
Supervised Seeded Iterated Learning for Interactive Language Learning (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has focused on word-based conversational agents that tend to invent their language rather than leveraging natural language. |
| Approach: | They propose two methods to counter language drift by combining S2P and Seeded Iterated Learning to minimize their weaknesses. |
| Outcome: | The proposed methods reduce late-stage training collapses and higher negative likelihood when evaluated on human corpus. |
Learning Natural Language Generation with Truncated Reinforcement Learning (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches to train conditional languagemodels without supervised learning fail to scale to large action spaces, thus allowing to train a language agent by only interacting with its environment without any task-specific prior knowledge. |
| Approach: | They propose an original approach to train conditional languagemodels without supervised learning by only using reinforcement learning. |
| Outcome: | The proposed approach avoids the dependency to labelled datasets and reduces pretrained policy flaws such as language or exposure biases. |
Countering Reward Over-Optimization in LLM with Demonstration-Guided Reinforcement Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches address ROO by adding KL regularization, requiring computationally expensive hyperparameter tuning. |
| Approach: | They propose a reinforcement learning approach that leverages human demonstrations and a reward model to recalibrate the reward objective. |
| Outcome: | The proposed approach achieves comparable performance to carefully tuned baselines while mitigating ROO in three RL language tasks. |
Contrastive Policy Gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashion (2024.emnlp-main)
Copied to clipboard
Yannis Flet-Berliac, Nathan Grinsztajn, Florian Strub, Eugene Choi, Bill Wu, Chris Cremer, Arash Ahmadian, Yash Chandak, Mohammad Azar, Olivier Pietquin, Matthieu Geist
| Challenge: | Reinforcement Learning (RL) is a method used to fine tune Large Language Models (LLMs) using a reward model trained from preference data to better align with human judgment. |
| Approach: | They propose a Reinforcement Learning (RL) algorithm that can estimate the optimal policy even from off-policy data. |
| Outcome: | The proposed algorithm can estimate the optimal policy even from off-policy data. |