Papers with LADM
LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Long-context modeling has drawn more attention in the area of Large Language Models (LLMs). |
| Approach: | They propose a Long-context data selection framework with Attention-based Dependency Measurement which can efficiently identify high-quality long-contrast data from a large-scale, multi-domain pre-training corpus. |
| Outcome: | The proposed framework significantly boosts the performance of LLMs on multiple long-context tasks with only 1B tokens for continual training. |