Papers by Daiki Matsunaga
GDPO: Learning to Directly Align Language Models with Diversity Using GFlowNets (2024.emnlp-main)
Copied to clipboard
| Challenge: | Reinforcement learning with human feedback (RLHF) and its offline variant Direct Preference Optimization (DPO) are two of the most important methods for language model (LM) alignment. |
| Approach: | They propose to use a diversity-seeking RL algorithm called GFlowNet-DPO in an offline preference alignment setting to optimize a model's behavior. |
| Outcome: | Empirical results show that the proposed algorithm generates far more diverse responses than the baseline methods and is still relatively aligned with human values in dialog generation and summarization tasks. |