Papers with PosGen
Resonance RoPE: Improving Context Length Generalization of Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have demonstrated their potential across a wide spectrum of natural language processing tasks. |
| Approach: | They propose a novel approach to narrow the generalization gap in TSTL scenarios by refining the interpolation of RoPE features for OOD positions. |
| Outcome: | The proposed approach improves performance without additional online computational costs on train-short-test-long scenarios. |