Papers by Raluca Popa
LLoCO: Learning Long Contexts Offline (2024.emnlp-main)
Copied to clipboard
Sijun Tan, Xiuyu Li, Shishir G Patil, Ziyang Wu, Tianjun Zhang, Kurt Keutzer, Joseph Gonzalez, Raluca Popa
| Challenge: | Large language models are still unable to handle long contexts due to the quadratic computational and memory overhead of the self-attention mechanism and the substantial KV cache sizes during generation. |
| Approach: | They propose a method to learn contexts offline through context compression and in-domain parameter-efficient finetuning with LoRA. |
| Outcome: | The proposed model outperforms in-context learning while using 30 fewer tokens during inference and significantly reduces the cost of long document question answering. |