Papers by Mete Ozay
K-Merge: Online Continual Merging of Adapters for On-device Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are powerful general-purpose models that can be adapted to a wide range of problem types in many languages. |
| Approach: | They propose a method for on-device online continual merging to integrate new LoRAs when a new one becomes available. |
| Outcome: | The proposed approach outperforms other methods while adhering to storage budget constraints. |
On-device System of Compositional Multi-tasking in Large Language Models (2025.emnlp-industry)
Copied to clipboard
Ondrej Bohdal, Konstantinos Theodosiadis, Asterios Mpatziakas, Dimitrios Filippidis, Iro Spyrou, Christos Zonios, Anastasios Drosou, Dimosthenis Ioannidis, Kyenghun Lee, Jijoong Moon, Hyeonmok Ko, Mete Ozay, Umberto Michieli
| Challenge: | Existing approaches to generative AI for large language models struggle when executing complex tasks simultaneously. |
| Approach: | They propose a novel approach tailored specifically for compositional multi-tasking scenarios . they add a learnable projection layer on top of the combined summarization and translation adapters. |
| Outcome: | The proposed approach performs well and is fast in both cloud-based and on-device implementations. |
HydraOpt: Navigating the Efficiency-Performance Trade-off of Adapter Merging (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods that produce a fixed trade-off between storage size and performance are often ineffective due to the growing size of large language models. |
| Approach: | They propose a model merging technique that capitalizes on similarities between low-rank adapters to reduce storage costs and improve performance. |
| Outcome: | The proposed method significantly reduces storage size (48% reduction) while outperforms existing merging techniques in terms of performance (0.2-1.8% drop). |
Multi-Task Pre-Finetuning of Lightweight Transformer Encoders for Text Classification and NER (2025.emnlp-industry)
Copied to clipboard
| Challenge: | nave multitask pre-finetuning introduces conflicting optimization signals that degrade overall performance. |
| Approach: | They propose a framework that enables a single shared encoder backbone with modular adapters. |
| Outcome: | The proposed framework achieves comparable performance to individual pre-finetuning while meeting practical deployment constraint. |
Decoding Text Spans for Efficient and Accurate Named-Entity Recognition (2026.acl-industry)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is a key component in industrial information extraction pipelines, where systems must satisfy strict latency and throughput constraints in addition to strong accuracy. |
| Approach: | They propose a span-based NER framework that can be used to compute span representations at the final transformer stage, avoiding redundant computation in earlier layers. |
| Outcome: | The proposed framework matches competitive baselines while improving throughput and reducing computational cost. |
A Study of Parameter Efficient Fine-tuning by Learning to Efficiently Fine-Tune (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for parameter-efficient fine-tuning (PEFT) are limited due to the need for increased computational resources. |
| Approach: | They propose a method to learn PEFT parameters from data by projecting high dimensional parameters onto low dimensional parameter manifolds or identifying PEFT parametrically. |
| Outcome: | The proposed method can be used to identify PEFT parameters on GLUE tasks. |
LoRA-Guard: Parameter-Efficient Guardrail Adaptation for Content Moderation of Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing model-based guardrails have not been designed for resource-constrained computational portable devices such as mobile phones. |
| Approach: | They propose a parameter-efficient guardrail adaptation method that relies on knowledge sharing between LLMs and guardrail models to adapt to content moderation tasks. |
| Outcome: | The proposed method outperforms existing guardrail methods with lower parameter overhead and higher accuracy on the generative task. |
Efficient Compositional Multi-tasking for On-device Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Adapter parameters provide a mechanism to modify the behavior of machine learning models and have gained significant popularity in the context of large language models (LLMs). |
| Approach: | They propose a benchmark for text-based compositional multi-tasking where multiple tasks are executed simultaneously. |
| Outcome: | The proposed method is optimized for on-device applications where computational resources are limited. |
Model Merging and Safety Alignment: One Bad Model Spoils the Bunch (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for merging large language models often overlook safety alignment during merging, leading to misaligned models. |
| Approach: | They propose to combine safety and domain-specific data to optimize model merging techniques . they propose to use this data to maximize model alignment . |
| Outcome: | The proposed method allows for models that excel in both domain expertise and alignment. |