Papers with YaRN
Resonance RoPE: Improving Context Length Generalization of Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have demonstrated their potential across a wide spectrum of natural language processing tasks. |
| Approach: | They propose a novel approach to narrow the generalization gap in TSTL scenarios by refining the interpolation of RoPE features for OOD positions. |
| Outcome: | The proposed approach improves performance without additional online computational costs on train-short-test-long scenarios. |
Extending Audio Context for Long-Form Understanding in Large Audio-Language Models (2026.eacl-long)
Copied to clipboard
Yuatyong Chaichana, Pittawat Taveekitworachai, Warit Sirichotedumrong, Potsawee Manakul, Kunat Pipatanakul
| Challenge: | Prior work has introduced context-extension methods (e.g. YaRN) on unimodal LLMs, yet their application to LALMs remains unexplored. |
| Approach: | They propose a training-free, modality-decoupled extension method that modifies only audio token positions, leaving text positions intact to preserve the base LLM’s text capabilities. |
| Outcome: | The proposed method outperforms the original models across wide range of settings and provides significant performance improvement on long audio of unseen lengths. |
PSC: Extending Context Window of Large Language Models via Phase Shift Calibration (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large-scale language models (LLMs) have shown impressive results across a variety of tasks. |
| Approach: | They propose a module for calibrating the frequencies predefined by existing methods . they conducted extensive experiments across multiple models and tasks . |
| Outcome: | The proposed method reduces perplexity as the context window size is varied from 16k to 32k and up to 64k. |
Beyond Binary Preferences: Semi-Online Label-Free GRACE-KTO with Group-Wise Adaptive Calibration for High-Quality Long-Text Generation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Generating high-quality long-text remains challenging for Large Language Models (LLMs), as conventional supervised fine-tuning fails to ensure overall quality due to its teacher-forcing nature. |
| Approach: | They propose a semi-online framework that transforms KTO’s binary signals into dynamically calibrated intra-group rewards. |
| Outcome: | The proposed framework transforms binary signals into dynamically calibrated intra-group rewards. |