Sci-LoRA: Mixture of Scientific LoRAs for Cross-Domain Lay Paraphrasing (2025.findings-acl)
Copied to clipboard
| Challenge: | Lay paraphrasing aims to make scientific information accessible to non-experts . existing studies focus on a single domain, such as biomedicine . |
| Approach: | a new lay paraphrasing model leverages a mixture of LoRAs fine-tuned on multiple scientific domains. |
| Outcome: | a new model outperforms state-of-the-art large language models in lay paraphrasing . the model can adjust the impact of different domains without explicit labels . |
Similar Papers
LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks (2024.acl-long)
Copied to clipboard
| Challenge: | LoRA-Flow uses lightweight modules to customize large language models for downstream tasks . previous work on LoRA combination relied on task-level weights for each involved LoRA . |
| Approach: | They propose a LoRA-Flow approach that uses dynamic weights to adjust the impact of different LoRAs. |
| Outcome: | The proposed method outperforms baselines with task-level weights on six generative tasks. |
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling (2026.acl-long)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning method for large language models. |
| Approach: | They propose a drop-in extension that reparameterizes a rank-rtot update as a sum of K *static* low-rank experts. |
| Outcome: | Experiments on reasoning and knowledge-intensive benchmarks show consistent gains over matched-budget LoRA. |
Making Science Simple: Corpora for the Lay Summarisation of Scientific Literature (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing datasets for lay summarisation are limited in size and scope, hindering the development of data-driven approaches. |
| Approach: | They propose to use two new datasets for the lay summarisation of biomedical research articles to characterise their lay summaries. |
| Outcome: | The proposed datasets are compared with existing datasets and show they can be leveraged to support different audiences and applications. |
Mixture-of-Subspaces in Low-Rank Adaptation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Using a subspace-inspired Low-Rank Adaptation method, large language models can be optimized for downstream tasks using parameter-efficient finetuning. |
| Approach: | They propose a subspace-inspired Low-Rank Adaptation method that decomposes LoRA weights into two subspaces and merges them into the frozen original weight. |
| Outcome: | The proposed method outperforms LoRA on commonsense reasoning, visual instruction tuning, and subject-driven text-to-image generation tasks. |
Mixture-of-LoRAs: An Efficient Multitask Tuning Method for Large Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Instruction Tuning has the potential to stimulate or enhance specific capabilities of large language models. |
| Approach: | They propose a mixture-of-LoRAs architecture which is a parameter-efficient tuning method designed for multi-task learning with LLMs. |
| Outcome: | The proposed method can be iteratively adapted to a new domain, enabling quick domain-specific adaptation. |
LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wild (2024.findings-acl)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) is an effective yet efficient solution for fine-tuning large language models. |
| Approach: | They propose a low-rank Adaptation framework that retrieves and composes multiple LoRAs according to input prompts. |
| Outcome: | Experimental results show that LoraRetriever outperforms baselines in terms of performance and versatility. |
LoRA Soups: Merging LoRAs for Practical Skill Composition Tasks (2025.coling-industry)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) is a popular technique for parameter-efficient fine-tuning of Large Language Models. |
| Approach: | They propose to combine LoRA modules to achieve skill composition . they propose to use concatenation of LoRAs to optimize weights for different LoRA training . |
| Outcome: | The proposed model outperforms existing models and data- merging techniques on math-word problems and domain-specialized corpora. |
MoDULA: Mixture of Domain-Specific and Universal LoRA for Multi-Task Learning (2024.emnlp-main)
Copied to clipboard
Yufei Ma, Zihan Liang, Huangyu Dai, Ben Chen, Dehong Gao, Zhuoran Ran, Wang Zihan, Linbo Jin, Wen Jiang, Guannan Zhang, Xiaoyan Cai, Libin Yang
| Challenge: | Recent advances in open-source Large Language Models (LLMs) have achieved notable successes in natural language processing. |
| Approach: | They propose a Parameter Efficient Fine-Tuning paradigm for improved fine-tuning and parameter efficiency in multi-task learning. |
| Outcome: | The proposed model outperforms existing methods on multi-task learning while reducing training costs by over 80% without losing general capability. |
SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature (2025.emnlp-main)
Copied to clipboard
David Wadden, Kejian Shi, Jacob Morrison, Alan Li, Aakanksha Naik, Shruti Singh, Nitzan Barzilay, Kyle Lo, Tom Hope, Luca Soldaini, Shannon Zejiang Shen, Doug Downey, Hannaneh Hajishirzi, Arman Cohan
| Challenge: | ScIRIFF is the only entirely expert-written instruction-following dataset for scientific literature understanding . it features complex instructions with long input contexts, detailed task descriptions, and structured outputs. |
| Approach: | They present a dataset of 137K instruction-following instances for training and evaluation . they finetuned large language models using a mix of general domain and ScIRIFF instructions . |
| Outcome: | The proposed dataset shows that on nine out-of-distribution held-out tasks, the model performs better than baselines trained on general domain instructions. |
Multi-Source Domain Adaptation with Mixture of Experts (D18-1)
Copied to clipboard
| Challenge: | Existing methods for domain adaptation from multiple sources are designed to transfer supervision from a single source domain. |
| Approach: | They propose to capture the relationship between a target example and different source domains by a point-to-set metric. |
| Outcome: | The proposed method outperforms baselines and can handle negative transfer. |