Papers with hypernetwork
HYPEROFA: Expanding LLM Vocabulary to New Languages via Hypernetwork-Based Embedding Initialization (2025.acl-srw)
Copied to clipboard
| Challenge: | Pre-trained language models exhibit suboptimal performance on mid- and low-resource languages due to limited exposure to these languages during pre-training. |
| Approach: | They propose a similarity-based subword embedding initialization heuristic that introduces new tokens specific to target languages, initializes their embedders, and applies continual pre-training on target-language data. |
| Outcome: | The proposed method outperforms random initialization baseline and matches or exceeds OFA in both continual pre-training convergence and downstream task performance. |
Text2Model: Text-based Model Induction for Zero-shot Image Classification (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to zero-shot learning are limited in two ways: Query-dependence and richness of language description. |
| Approach: | They propose a task-agnostic approach to image classification using only text descriptions . they train a hypernetwork that receives class descriptions and outputs a multi-class model . |
| Outcome: | The proposed approach generates non-linear classifiers, handles rich textual descriptions, and may be adapted to produce lightweight models efficient enough for on-device applications. |
HyperEdit: Unlocking Instruction-based Text Editing in LLMs via Hypernetworks (2026.findings-acl)
Copied to clipboard
Yiming Zeng, Jinghan Cao, Zexin Li, Wanhao Yu, Zhankai Ye, Dawei Xiang, Ting Hua, Xin Liu, Shangqian Gao, Tingting Yu
| Challenge: | Existing approaches treat instruction-based text editing as a generic text generation problem. Existing methods either over-edit or fail to apply modifications consistently. |
| Approach: | They propose a framework that processes each editing request to best align with it. |
| Outcome: | The proposed framework achieves 9% improvement over the state-of-the-art model. |
Hyper-CL: Conditioning Sentence Representations with Hypernetworks (2024.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to sentence embeddings do not capture fine-grained semantics of sentences. |
| Approach: | They propose a method that integrates hypernetworks with contrastive learning to generate conditioned sentence representations. |
| Outcome: | The proposed method narrows the performance gap with the bi-encoder architecture while maintaining the time efficiency characteristic of the tri-encoding approach. |
Learning to Generate Task-Specific Adapters from Task Description (2021.acl-short)
Copied to clipboard
| Challenge: | Pre-trained text-to-text transformers have achieved impressive performance across a range of NLP tasks, such as question answering and commonsense reasoning. |
| Approach: | They propose a framework that improves text-to-text transformer’s generalization ability to unseen tasks by training a hypernetwork to generate task-specific adapters from task descriptions. |
| Outcome: | Experiments on ZEST and a synthetic SQuAD dataset show that Hypter improves upon fine-tuning baselines. |
Awakening Augmented Generation: Learning to Awaken Internal Knowledge of Large Language Models for Question Answering (2025.coling-main)
Copied to clipboard
| Challenge: | Recent studies indicate that Large Language Models model rich knowledge, but it is often not activated and awakened. |
| Approach: | They propose a framework that leverages richer context to enhance question answering . Explicit awakening fine-tunes a context generator to create a synthetic, compressed document that functions as symbolic context. |
| Outcome: | The proposed framework mimics the human ability to answer questions using only thinking and recalling to compensate for knowledge gaps. |
Hyperdecoders: Instance-specific decoders for multi-task NLP (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work in NLP has examined the performance of large pretrained transformer-based models in multi-task settings, where a single model is evaluated on multiple tasks simultaneously. |
| Approach: | They propose a method for multi-tasking using a hypernetwork conditioned on the output of an encoder to generate a unique decoder adaptation for every input instance. |
| Outcome: | The proposed method outperforms previous methods for efficient multi-task fine-tuning and maps from encoder representations to output labels. |
GenPoE: Generative Passage-level Mixture of Experts for Knowledge Enhancement of LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | GenPoE is a passage-level mixture of experts for enhancing knowledge of large language models. |
| Approach: | They propose a novel “generative” passage-level mixture of experts (MoEs) that takes in-context retrieved passages and generates their “expert” parameters. |
| Outcome: | The proposed system is based on a novel hypernetwork which takes in-context retrieved passages and generates their "expert'' parameters. |
LayerConnect: Hypernetwork-Assisted Inter-Layer Connector to Enhance Parameter Efficiency (2022.coling-1)
Copied to clipboard
| Challenge: | Existing parameter-efficient methods focus on reducing trainable parameters but neglect the inference speed, which limits the ability to deploy PLMs. |
| Approach: | They propose to use a hypernetwork-assisted inter-layer connector to enhance inference efficiency by tuning parameters inside a linear connector between two Transformer layers. |
| Outcome: | The proposed model reduces model parameters to 11.75% while preserving performance degradation to less than 5%. |
Prototype-based HyperAdapter for Sample-Efficient Multi-task Tuning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pre-trained language models are limited to low-data regimes and require learning different modules to adapt to diverse tasks. |
| Approach: | They propose a framework for parameter-efficient fine-tuning that trains modules per task . they use an instance-dense retriever and a prototypical hypernetwork to generate conditional modules . |
| Outcome: | The proposed framework outperforms existing methods on multi-task learning and few-shot transfer learning. |
AdaTag: Multi-Attribute Value Extraction from Product Profiles with Adaptive Decoding (2021.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to extract product attribute values are limited by knowledge sharing across different attributes. |
| Approach: | They propose to use adaptive decoding to handle extraction of product attribute values by parameterizing the decoder with pretrained attribute embeddings, through a hypernetwork and a Mixture-of-Experts module. |
| Outcome: | The proposed model is able to handle multiple attributes without sharing the entire network parameters across all attributes. |
Meta-Learning with Variational Semantic Memory for Word Sense Disambiguation (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods for word sense disambiguation (WSD) lack large annotated datasets with sufficient coverage of words . performance of such methods lags behind fully-supervised methods . a meta-learning model is proposed to solve this problem . |
| Approach: | They propose a model of semantic memory for supervised word sense disambiguation using meta-learning. |
| Outcome: | The proposed model improves performance in few-shot WSD and produces meaning prototypes that capture similar senses of distinct words. |
Meta-Tool: Efficient Few-Shot Tool Adaptation for Small Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Using a Llama-3.2-3B-Instruct backbone, we evaluate four adaptation mechanisms across four benchmarks: Gorilla APIBench, Spider 2.0, WebArena, and InterCode. |
| Approach: | They compare hypernetwork-based LoRA adaptation against carefully designed few-shot prompting in a controlled experiment . they find that few- shot prompting contributes +21.5% to performance and documentation contributes 0% . |
| Outcome: | The hypernetwork-based LoRA adaptation provides no measurable improvement over few-shot prompting alone. |
Hyper-X: A Unified Hypernetwork for Multi-Task Multilingual Transfer (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing multilingual models cannot fully leverage training data when it is available in different task-language combinations. |
| Approach: | They propose a single hypernetwork that unifies multi-task and multilingual learning with efficient adaptation. |
| Outcome: | The proposed model achieves the best or competitive gain when a mixture of multiple resources is available while being significantly more efficient than existing models. |
Towards Anytime Fine-tuning: Continually Pre-trained Language Models with Hypernetwork Prompts (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Continual pre-training has been used for a multitude of domains and tasks . a continually pre-trained model can show a non-decreasing performance on unseen domains . |
| Approach: | They propose a method that generates domain-specific prompts by agreement and disagreement losses. |
| Outcome: | The proposed method achieves improvements of 3.57% and 3.4% on two real-world datasets. |
HyperLoRA: Efficient Cross-task Generalization via Constrained Low-Rank Adapters Generation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to adapt pre-trained language models (PLMs) to emerging tasks are costly and inefficient. |
| Approach: | They propose a meta-network that generates task-specific weights without any optimization. |
| Outcome: | The proposed approach has flexible generalization ability and superior performance over hypenetworks. |
(Almost) Free Modality Stitching of Foundation Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multi-modal foundation models often use modality-specific (uni-modal) models as sub-components, which are stitched together via a connector module. |
| Approach: | They propose a framework that allows for optimal uni-modal model selection and connector training by leveraging hypernetworks. |
| Outcome: | The proposed framework reduces the cost of searching for the best performing uni-modal model pair by 10 while matching the ranking and trained connector performance across diverse multi-modal benchmarks. |
Instant Personalized Large Language Model Adaptation via Hypernetwork (2026.acl-long)
Copied to clipboard
Zhaoxuan Tan, Zixuan Zhang, Haoyang Wen, Zheng Li, Rongzhi Zhang, Pei Chen, Fengran Mo, Zheyuan Liu, Qingkai Zeng, Qingyu Yin, Meng Jiang
| Challenge: | Existing parameter-efficient fine-tuning methods require training a separate adapter for each user, making them computationally expensive and impractical for real-time updates. |
| Approach: | They propose a scalable framework that maps a user's profile directly to a full set of adapter parameters. |
| Outcome: | The proposed framework outperforms prompt-based personalization and OPPU while using substantially fewer computational resources at deployment. |
HyperAdaLoRA: Accelerating LoRA Rank Allocation During Training via Hypernetworks without Sacrificing Performance (2026.findings-acl)
Copied to clipboard
Hao Zhang, Zhenjia Li, Yifan Gao, Xi Xiao, Heng Zhang, Shuyang Zhang, null Xiaoxincc, Bo Huang, Yuhang Wu, Tianyang Wang, Hao Xu
| Challenge: | Low-Rank Adaptation (LoRA) assumes a uniform rank r for each incremental matrix, not accounting for the varying significance of weight matrices across modules and layers. |
| Approach: | They propose a framework that allows for faster convergence of low-rank adaptive models . they use a hypernetwork to prune the outputs of the hypernetworks to generate parameters . |
| Outcome: | The proposed framework accelerates convergence of AdaLoRA by leveraging a hypernetwork. |