EpLSA: Synergy of Expert-prefix Mixtures and Task-Oriented Latent Space Adaptation for Diverse Generative Reasoning (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing models for diverse generative reasoning struggle to generate multiple unique and plausible results. |
| Approach: | They propose a model based on expert-prefix mixtures and task-oriented latent space adaptation for diverse generative reasoning. |
| Outcome: | The proposed model outperforms existing models on three types of generative reasoning tasks. |
Similar Papers
Eliciting and Understanding Cross-task Skills with Task-level Mixture-of-Experts (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained transformer models are capable of multitasking on diverse NLP tasks, but little is known about how multitaskability and cross-task generalization is achieved. |
| Approach: | They propose to use a transformer-based mixture-of-expert model with a router component to choose among experts dynamically and flexibly. |
| Outcome: | The proposed models improve the average performance gain (ARG) metric by 2.6% when adapting to unseen tasks, and by 5.6% in zero-shot generalization settings. |
GenPoE: Generative Passage-level Mixture of Experts for Knowledge Enhancement of LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | GenPoE is a passage-level mixture of experts for enhancing knowledge of large language models. |
| Approach: | They propose a novel “generative” passage-level mixture of experts (MoEs) that takes in-context retrieved passages and generates their “expert” parameters. |
| Outcome: | The proposed system is based on a novel hypernetwork which takes in-context retrieved passages and generates their "expert'' parameters. |
An End-to-End Generative Architecture for Paraphrase Generation (D19-1)
Copied to clipboard
| Challenge: | Existing methods for generating paraphrases with linguistic knowledge are often domain specific and hard to scale, or yield inferior results. |
| Approach: | They propose an end-to-end conditional generative architecture for generating paraphrases via adversarial training which does not depend on extra linguistic information. |
| Outcome: | The proposed method outperforms existing models on automatic metrics and human evaluations on four public datasets. |
Mixture-of-Linguistic-Experts Adapters for Improving and Interpreting Pre-trained Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | In recent years, pre-trained language models have become the de facto instrument for the field of natural language processing (NLP). |
| Approach: | They propose a method that injects linguistic structures into pre-trained language models in the parameter-efficient fine-tuning setting. |
| Outcome: | The proposed approach outperforms state-of-the-art methods with a comparable number of parameters. |
MixLoRA-DSI: Dynamically Expandable Mixture-of-LoRA Experts for Rehearsal-Free Generative Retrieval over Dynamic Corpora (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to update model-based indexes with new documents are expensive and require expensive retraining. |
| Approach: | They propose a framework that combines an expandable mixture of Low-Rank Adaptation experts with a layer-wise out-of-distribution-driven expansion strategy. |
| Outcome: | Experiments on NQ320k and MS MARCO Passage show that the proposed framework outperforms full-model update baselines with minimal parameter overhead and substantially lower training costs. |
A Closer Look into Mixture-of-Experts in Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Mixture-of-experts (MoE) architectures are gaining increasing attention for their unique properties and remarkable performance. |
| Approach: | They propose a mixture-of-experts architecture that allows for model scaling without sacrificing computational efficiency. |
| Outcome: | The proposed model increases model size without sacrificing computational efficiency . the proposed model is modular and can be used by a broad spectrum of practitioners . |
Towards Adaptive Prefix Tuning for Parameter-Efficient Language Model Fine-tuning (2023.acl-short)
Copied to clipboard
| Challenge: | Parameter-efficient fine-tuning only optimizes a few task-specific parameters with frozen pre-trained model. |
| Approach: | They propose to optimize a prefix vector inserted into Transformer layers to optimize the prefix . they propose to use a gate mechanism to adjust the prefixed to each layer . |
| Outcome: | The proposed approach improves on the SuperGLUE and NER datasets. |
AEA: Adaptive Expert Allocation Improves Sentence Embeddings from Mixture-of-Experts LLM (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods to improve embeddings from Mixture-of-Experts models allocate a fixed number of experts uniformly across all layers and tokens, ignoring inter-layer and inter-token heterogeneity. |
| Approach: | They propose an Adaptive Expert Allocation framework that performs layer-wise and token-wise expert allocation to enhance embedding quality. |
| Outcome: | The proposed method improves embedding quality across multiple MoE models. |
A Semantically Consistent and Syntactically Variational Encoder-Decoder Framework for Paraphrase Generation (2020.coling-main)
Copied to clipboard
| Challenge: | Paraphrase generation is a longstanding problem in natural language processing (NLP) Neural network-based methods have shown great progress on paraphrase generation. |
| Approach: | They propose a framework that integrates variational inference on a target-related latent variable to introduce the diversity. |
| Outcome: | The proposed framework outperforms baseline models on the metrics based on n-gram matching and semantic similarity, and it can generate multiple different paraphrases by assembling different syntactic variables. |
Selective Prefix Tuning for Pre-trained Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pre-trained models are time-consuming and memory-inefficient. |
| Approach: | They propose a method that inserts learnable vectors into each Transformer layer . they propose SL to encourage diversity in prefix tokens . |
| Outcome: | Extensive experiments validate the effectiveness of Prefix Tuning in sentence and token classification tasks. |