Papers with Product-of-Experts
Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results prove that language models can learn from human feedback better, irrespective of sequence length . emergence of length bias often induces the model to favor longer outputs . |
| Approach: | They propose to separate reward modeling from the influence of sequence length by using the Product-of-Experts technique. |
| Outcome: | The proposed approach shows that language models perform better regardless of sequence length . the main expert is focused on understanding human intents, while the biased expert targets the identification and capture of length bias. |