Papers by Olivier Pietquin
Supervised Seeded Iterated Learning for Interactive Language Learning (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has focused on word-based conversational agents that tend to invent their language rather than leveraging natural language. |
| Approach: | They propose two methods to counter language drift by combining S2P and Seeded Iterated Learning to minimize their weaknesses. |
| Outcome: | The proposed methods reduce late-stage training collapses and higher negative likelihood when evaluated on human corpus. |
Learning Natural Language Generation with Truncated Reinforcement Learning (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches to train conditional languagemodels without supervised learning fail to scale to large action spaces, thus allowing to train a language agent by only interacting with its environment without any task-specific prior knowledge. |
| Approach: | They propose an original approach to train conditional languagemodels without supervised learning by only using reinforcement learning. |
| Outcome: | The proposed approach avoids the dependency to labelled datasets and reduces pretrained policy flaws such as language or exposure biases. |
Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedback (2023.acl-long)
Copied to clipboard
Paul Roit, Johan Ferret, Lior Shani, Roee Aharoni, Geoffrey Cideron, Robert Dadashi, Matthieu Geist, Sertan Girgin, Leonard Hussenot, Orgad Keller, Nikola Momchev, Sabela Ramos Garea, Piotr Stanczyk, Nino Vieillard, Olivier Bachem, Gal Elidan, Avinatan Hassidim, Olivier Pietquin, Idan Szpektor
| Challenge: | Recent advances in abstractive summarization systems produce factually inconsistent text . this is emphasized in tasks like summarizing, which often produce inconsistent text with no input article . |
| Approach: | They use reinforcement learning to optimize for factual consistency and explore trade-offs . they use textual-entailment rewards to optimize the accuracy of the generated summaries . |
| Outcome: | The proposed method improves faithfulness, salience and conciseness of the generated summaries. |
Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs (2024.acl-long)
Copied to clipboard
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, Sara Hooker
| Challenge: | Proximal Policy Optimization (PPO) is used for RLHF but requires high computational cost and sensitive hyperparameter tuning. |
| Approach: | They propose to use Proximal Policy Optimization to align large language models to human preferences. |
| Outcome: | The proposed method preserves and even increases performance while preserving the motivational principles that led to the development of PPO. |
Countering Reward Over-Optimization in LLM with Demonstration-Guided Reinforcement Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches address ROO by adding KL regularization, requiring computationally expensive hyperparameter tuning. |
| Approach: | They propose a reinforcement learning approach that leverages human demonstrations and a reward model to recalibrate the reward objective. |
| Outcome: | The proposed approach achieves comparable performance to carefully tuned baselines while mitigating ROO in three RL language tasks. |
Contrastive Policy Gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashion (2024.emnlp-main)
Copied to clipboard
Yannis Flet-Berliac, Nathan Grinsztajn, Florian Strub, Eugene Choi, Bill Wu, Chris Cremer, Arash Ahmadian, Yash Chandak, Mohammad Azar, Olivier Pietquin, Matthieu Geist
| Challenge: | Reinforcement Learning (RL) is a method used to fine tune Large Language Models (LLMs) using a reward model trained from preference data to better align with human judgment. |
| Approach: | They propose a Reinforcement Learning (RL) algorithm that can estimate the optimal policy even from off-policy data. |
| Outcome: | The proposed algorithm can estimate the optimal policy even from off-policy data. |