Papers by Eran Malach
LoRA Soups: Merging LoRAs for Practical Skill Composition Tasks (2025.coling-industry)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) is a popular technique for parameter-efficient fine-tuning of Large Language Models. |
| Approach: | They propose to combine LoRA modules to achieve skill composition . they propose to use concatenation of LoRAs to optimize weights for different LoRA training . |
| Outcome: | The proposed model outperforms existing models and data- merging techniques on math-word problems and domain-specialized corpora. |