MIXTURE-OF-EXPERTS

NeKo: Cross-Modality Post-Recognition Error Correction with Tasks-Guided Mixture-of-Experts Language Model (2025.acl-industry)

Challenge:	MixtureKit is a modular open-source framework for constructing, training, and analyzing Mixture-of-Experts (MoE) models from arbitrary pre-trained or fine-tuned checkpoints.
Approach:	They propose a modular open-source framework for constructing, training, and analyzing Mixture-of-Experts (MoE) models from arbitrary pre-trained or fine-tuned checkpoints.
Outcome:	Experiments on multilingual code-switched (Arabic–Latin) show that BTX models built with MixtureKit outperform dense baselines across multiple benchmarks.

Copied to clipboard

Yen-Ting Lin, Zhehuai Chen, Piotr Zelasko, Zhen Wan, Xuesong Yang, Zih-Ching Chen, Krishna C Puvvada, Ke Hu, Szu-Wei Fu, Jun Wei Chiu, Jagadeesh Balam, Boris Ginsburg, Yu-Chiang Frank Wang, Chao-Han Huck Yang

Mixture of Heterogeneous Grouped Experts for Language Modeling (2026.acl-industry)

Challenge:	Existing methods to train a model on a mixture of domain datasets require separate correction language models.
Approach:	They propose a multi-task correction MoE that trains experts to become an "expert" of speech-to-text, language-totext and vision-to text datasets by learning to route each dataset’s tokens to its mapped expert.
Outcome:	The proposed model outperforms GPT-3.5 and Claude-3.5-Sonnet on the Open ASR Leaderboard and reaches an average relative 5.0% WER reduction and substantial improvements in BLEU scores.

Copied to clipboard

Zhicheng Ma, Xiang Liu, Zhaoxiang Liu, Ning Wang, Yi Shen, Kai Wang, Shuming Shi, Shiguo Lian

Accelerating Dense LLMs via L0-regularized Mixture-of-Experts (2025.acl-short)

Challenge:	Large Language Models (LLMs) based on Mixture-of-Experts (MoE) enforce uniform expert sizes, creating a rigidity that fails to align computational costs with varying token-level complexity.
Approach:	They propose a mixture of heterogeneous grouped experts (MoHGE) that allows for flexible, resource-aware expert combinations.
Outcome:	The proposed model matches the performance of existing Mixture-of-Experts architectures while maintaining balanced GPU utilization.

Copied to clipboard

Zhenyu Zhang, JiuDong Yang, Taozhaowen Taozhaowen, Meng Chen

Bag of Tricks for Sparse Mixture-of-Experts: A Benchmark Across Reasoning, Efficiency, and Safety (2025.findings-emnlp)

Challenge:	Existing methods for accelerating large language models (LLMs) suffer from slow and costly inference.
Approach:	They propose a lightweight MoE approach using cluster confusion matrix and dynamic batching to accelerate dense LLMs.
Outcome:	The proposed method achieves 2.5x speedup over dense models while maintaining competitive performance.

Copied to clipboard

Mufan Qiu, Zheyu Shen, Pingzhi Li, Ang Li, Tianlong Chen

Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models (2024.emnlp-main)

Challenge:	Existing benchmarks focus on isolated aspects of MoE, with conflicting conclusions . a lack of consensus on optimal design choices is limiting to specific aspects of the model.
Approach:	They propose to evaluate two popular MoE backbones across four dimensions of design choices . they find token-level routing and z-loss regularization improve reasoning performance .
Outcome:	The proposed framework evaluates two popular MoE backbones on over eight metrics.

Copied to clipboard

Zihan Wang, Deli Chen, Damai Dai, Runxin Xu, Zhuoshu Li, Yu Wu

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models (2024.acl-long)

Challenge:	Existing studies on parameter-efficient fine-tuning (PEFT) for dense-architecture LLMs are lacking.
Approach:	They propose an expert-specialized fine-tuning method that tunes the experts most relevant to downstream tasks while freezing the other experts.
Outcome:	The proposed method matches or surpasses full-parameter fine-tuning.

Copied to clipboard

Damai Dai, Chengqi Deng, Chenggang Zhao, R.x. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y.k. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, Wenfeng Liang

Expert Calibration Lens for Pruning Mixture of Experts (2026.acl-demo)

Challenge:	Mixture-of-Experts (MoE) architectures face challenges in ensuring expert specialization . despite the promising performance, scaling language models to an extremely large scale is associated with exceedingly high computational costs.
Approach:	They propose an architecture that allows for ultimate expert specialization by segmenting experts into mN ones and activating mK from them.
Outcome:	The proposed architecture achieves comparable performance with GShard with 2B parameters and computation.

Copied to clipboard

Luis Frentzen Salim, Chia-Chun Wu, Tran Van Nhiem, Lun-Wei Ku, Yung-Hui Li

Challenge:	Expert pruning is a practical deployment technique for Mixture-of-Experts models . but its success depends heavily on the calibration set used for pruning .
Approach:	They propose a calibration tool that compares expert activations across datasets to predict calibration perturbations without running expensive pruning procedures.
Outcome:	The proposed system compares expert activations across datasets to predict calibration perturbations without running expensive pruning procedures.

Dynamic Data Mixing Maximizes Instruction Tuning for Mixture-of-Experts (2025.naacl-long)

Copied to clipboard

Tong Zhu, Daize Dong, Xiaoye Qu, Jiacheng Ruan, Wenliang Chen, Yu Cheng

Condensing Multilingual Knowledge with Lightweight Language-Specific Modules (2023.emnlp-main)

Challenge:	Mixture-of-Experts (MoE) models are constrained by their fixed model capacities when the number of tasks grows in instruction tuning.
Approach:	They propose to combine all training tasks and apply fixed sampling weights without considering the importance of different tasks as the model training state changes.
Outcome:	The proposed method can be used on knowledge & reasoning tasks and open-ended queries with limited training budget.

Copied to clipboard

Haoran Xu, Weiting Tan, Shuyue Li, Yunmo Chen, Benjamin Van Durme, Philipp Koehn, Kenton Murray

Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design (2026.acl-industry)

Challenge:	Existing methods to boost performance in multilingual models but scalability is difficult to manage.
Approach:	They propose a method that incorporates language-specific (LS) modules to boost model performance.
Outcome:	The proposed method outperforms state-of-the-art methods while outperforming existing methods.

Copied to clipboard

Junzhuo Li, Peijie Jiang, Changxin Tian, Jia Liu, Zhiqiang Zhang, Xuming Hu

Specialization through Collaboration: Understanding Expert Interaction in Mixture-of-Expert Large Language Models (2026.eacl-long)

Challenge:	a novel extension of neural scaling laws to Mixture-of-Experts models is proposed . a ratio of expert-attention compute is crucial for efficient MoE models .
Approach:	They propose an extension of neural scaling laws to Mixture-of-Experts (MoE) models . they define the ratio r as the fraction of total FLOPs per token dedicated to expert and attention layers .
Outcome:	The proposed model can be tuned beyond size and data with the proposed model.

Copied to clipboard

Yuanbo Tang, Naifan Zhang, Yan Tang, Meixuan Chen, Shuhan Huang, Tingyu Cao, Yang Li

When the Model Said ‘No Comment’, We Knew Helpfulness Was Dead, Honesty Was Alive, and Safety Was Terrified (2026.eacl-long)

Challenge:	Mixture-of-Experts (MoE) based large language models are popular for multitasking . however, whether each expert can specialize to a task remains unclear .
Approach:	They propose to use a dictionary learning approach to analyze expert collaboration mechanisms in MoE LLMs.
Outcome:	The proposed model outperforms existing methods by 2.5% while enabling 50% expert reduction.

Copied to clipboard

Gautam Siddharth Kashyap, Mark Dras, Usman Naseem

NeuronMoE: Efficient Cross-Lingual Extension via Neuron-Guided Mixture-of-Experts (2026.eacl-long)

Challenge:	Existing work uses SFT and MoE to align Large Language Models, but these work face challenges in multi-objective settings.
Approach:	They propose a framework that uses prompt-injected fine-tuning to extract axis-specific task features . it deploys a MoCaE module that calibrates expert routing using fractal and natural geometry .
Outcome:	The proposed framework achieves significant gains on Alpaca, BeaverTails, TruthfulQA and TruthfulQ with +171.5% win rate and +110.1% truthfulness-informativeness.

Copied to clipboard

Rongzhi Li, Hitomi Yanaka

Mixture of insighTful Experts (MoTE): The Synergy of Reasoning Chains and Expert Mixtures in Self-Alignment (2025.acl-long)

Challenge:	Existing approaches allocate experts based on layer-level similarity, yet language processing exhibits fine-grained specialization at individual neurons.
Approach:	They propose a method that analyzes language-specific neurons to guide expert allocation per layer based on cross-lingual neuron diversity.
Outcome:	The proposed method reduces the complexity of the model by 40% while matching the performance of the LayerMoE baseline.

Copied to clipboard

Zhili Liu, Yunhao Gou, Kai Chen, Lanqing Hong, Jiahui Gao, Fei Mi, Yu Zhang, Zhenguo Li, Xin Jiang, Qun Liu, James Kwok

Reasoning

Challenge:	Recent studies show that reasoning abilities contribute significantly to model safety, while integrating Mixture-of-Experts (MoE) architectures can further enhance alignment.
Approach:	They propose a framework that synergistically combines reasoning chains and expert mixtures to improve self-alignment.
Outcome:	The proposed framework improves model safety, jailbreak resistance, and over-refusal capabilities, achieving performance comparable to OpenAI’s state-of-the-art o1 model.

ModularMoE: Fast LLM Customization with Parameter-Sharing Mixture-of-Experts for Low-Resource Settings (2026.findings-acl)

Copied to clipboard

Jiaxing Liu, Qi Qi, Haifeng Sun, Dunjun Li, Zirui Zhuang, Bo He, Xiang Yang, Cong Liu, Jianxin Liao, Jingyu Wang

FoldMoE: Efficient Long Sequence MoE Training via Attention-MoE Pipelining (2025.acl-long)

Challenge:	Large Language Models impose significant computational and storage burdens on personal devices . existing customization approaches incur excessive computational costs or lead to suboptimal performance .
Approach:	They propose a training framework that converts pre-trained LLMs into parameter-sharing MoE models for lightweight deployment.
Outcome:	The proposed training framework outperforms state-of-the-art training frameworks at the same sparsity level while delivering up to 2.71 inference speedup.

Copied to clipboard

Guichao Zhu, Lintian Lei, Yuhao Qing, Yichao Fu, Fanxin Li, Dong Huang, Zekai Sun, Heming Cui

Beyond Spurious Signals: Debiasing Multimodal Large Language Models via Counterfactual Inference and Adaptive Expert Routing (2025.findings-emnlp)

Challenge:	Existing approaches to training LLMs with Mixture-of-Experts (MoE) architecture on long sequences are limited by the insufficient computation.
Approach:	They propose a MoE training system that enables token-level overlapping across entire Transformer blocks through novel attention-MoE pipelining.
Outcome:	The proposed system achieves 1.49x and 2.72x speedup over state-of-the-art token-level overlapping and non-overlapping baselines on GPT-MoE models with sequences up to 32K tokens.

Copied to clipboard

Zichen Wu, Hsiu-Yuan Huang, Yunfang Wu

Robustness in NLP

Challenge:	Multimodal Large Language Models (MLLMs) often rely on spurious correlations, undermining their robustness and generalization.
Approach:	They propose a causal mediation-based debiasing framework to address correlation bias in MLLMs . they distinguish core semantics from spurious textual and visual contexts using counterfactual examples .
Outcome:	The proposed framework surpasses existing state-of-the-art models on sarcasm detection and sentiment analysis tasks.

Analytical FFN-to-MoE Restructuring via Activation Pattern Analysis (2026.acl-long)

Copied to clipboard

Zehua Pei, Hui-Ling Zhen, Lancheng Zou, Xianzhi Yu, Wulong Liu, Sinno Jialin Pan, Mingxuan Yuan, Bei Yu

Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Models (2024.findings-emnlp)

Challenge:	Large language models (LLMs) are fast but require expensive pre-training . a new approach to scale large language models into MoEs reduces inference costs .
Approach:	They propose an analytical post-training framework that rapidly restructures FFNs into sparse MoE architectures using only a small calibration dataset.
Outcome:	The proposed framework outperforms existing methods on a small calibration dataset.

Copied to clipboard

Songtao Jiang, Tuo Zheng, Yan Zhang, Yeying Jin, Li Yuan, Zuozhu Liu

Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models (2026.acl-long)

Challenge:	Recent advances in multimodal large language models have seen remarkable progress for medical decision-making, however, they are designated for specific classification or generative tasks and require model training or finetuning on large-scale datasets with sizeable parameters and tremendous computing.
Approach:	They propose a framework that tackles discriminative and generative multimodal medical tasks using multimodal alignment, instruction tuning and routing.
Outcome:	The proposed model can achieve superior performance to or on par with state-of-the-art baselines while only requiring 30%-50% of activated model parameters.

Copied to clipboard

Xin Cheng, Wangding Zeng, Damai Dai, Qinyu Chen, Bingxuan Wang, Zhenda Xie, Kezhao Huang, Xingkai Yu, Zhewen Hao, Han Zhang, Yu-Kun Li, Huishuai Zhang, Dongyan Zhao, Wenfeng Liang

AID: Adaptive Integration of Detectors for Safe AI with Language Models (2025.naacl-long)

Challenge:	Mixture-of-Experts (MoE) scales capacity via conditional computation, but lacks knowledge lookup primitive.
Approach:	They propose a conditional memory instantiated via Deep Sparse Embedding (DSE) they propose 'u-shaped scaling law' that identifies optimal balance between MoE experts and DSE memory .
Outcome:	The proposed model outperforms an iso-parameter and isoFLOPs MoE baseline across knowledge and reasoning benchmarks and is infrastructure-efficient.

Copied to clipboard

Xinran Wang, Enmao Diao, Qi Le, Jie Ding, Ali Anwar

Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modules (2026.findings-eacl)

Challenge:	Large language models (LLMs) are increasingly used to generate human-like text, but safety concerns have emerged with the deployment of LLMs.
Approach:	They propose an approach that orchestrates the strengths of multiple pretrained detectors to ensure comprehensive effectiveness in diverse scenarios.
Outcome:	The proposed approach can improve the area under the curve (AUC) by 0.07 to 0.21, with a median of 0.12, compared to the best individual detectors developed for specific safety aspects.

Copied to clipboard

Yilun Liu, Yunpu Ma, Yuetian Lu, Shuo Chen, Zifeng Ding, Volker Tresp

Memory Augmented Language Models through Mixture of Word Experts (2024.naacl-long)

Challenge:	Existing Parameter-Efficient Fine-Tuning (PEFT) strategies that focus on specialized experts are not effective for Mixture-of-Experts (MoE).
Approach:	They propose to integrate a dynamic routing mechanism among specialized experts in Mixture-of-Experts (MoE) .
Outcome:	Extensive experiments on commonsense and math reasoning tasks validate the performance and efficiency of the proposed routed approach.

Copied to clipboard

Cicero Nogueira dos Santos, James Lee-Thorp, Isaac Noble, Chung-Ching Chang, David Uthus

Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models (2025.acl-long)

Challenge:	Increasing the parameter count of language models has been a primary driver of improved model quality, but increasing the model size also increases the cost of training and serving the model.
Approach:	They propose to decouple learning capacity and FLOPs by using a mixture-of-experts approach with large knowledge-rich vocabulary based routing functions.
Outcome:	The proposed model outperforms the T5 family of models with similar number of FLOPs on knowledge intensive tasks and similar performance to memory augmented approaches.

Copied to clipboard

Zihan Qiu, Zeyu Huang, Bo Zheng, Kaiyue Wen, Zekun Wang, Rui Men, Ivan Titov, Dayiheng Liu, Jingren Zhou, Junyang Lin

Programming Languages in NLP

Challenge:	Existing Mixture-of-Experts training frameworks use a micro-batch to calculate LBL . micro-batches are restricted to a single sequence, preventing expert specialization .
Approach:	They propose to use a global-batch to loosen the load balance constraint for MoEs models . they propose to synchronize fi across micro-batches and then use it to calculate the LBL .
Outcome:	The proposed global-batch LBL improves the domain specialization of experts . the micro-battery LBL is almost at the sequence level, and the router is pushed to distribute the token evenly .

Emergent Modularity in Pre-trained Transformers (2023.findings-acl)

Copied to clipboard

Zhengyan Zhang, Zhiyuan Zeng, Yankai Lin, Chaojun Xiao, Xiaozhi Wang, Xu Han, Zhiyuan Liu, Ruobing Xie, Maosong Sun, Jie Zhou

Mixture of Attention Heads: Selecting Attention Heads Per Token (2022.emnlp-main)

Challenge:	Existing studies on pre-trained Transformers show that they learn fine-grained neuron functions.
Approach:	They examine the presence of modularity in pre-trained Transformers . they focus on Mixture-of-Experts, a promising candidate for modularity .
Outcome:	The proposed structure stabilizes at the early stage, which is faster than neuron stabilization.

Copied to clipboard

Xiaofeng Zhang, Yikang Shen, Zeyu Huang, Jie Zhou, Wenge Rong, Zhang Xiong

MoLA: MoE LoRA with Layer-wise Expert Allocation (2025.findings-naacl)

Challenge:	Mixture-of-Experts (MoE) networks have been proposed as an efficient way to scale up model capacity and implement conditional computing.
Approach:	They propose a new architecture that combines multi-head attention with the MoE mechanism and a sparsely gated architecture that allows for faster computations.
Outcome:	The proposed architecture can scale up the number of attention heads and the number parameters while preserving computational efficiency.

Copied to clipboard

Chongyang Gao, Kezhen Chen, Jinmeng Rao, Ruibo Liu, Baochen Sun, Yawen Zhang, Daiyi Peng, Xiaoyuan Guo, Vs Subrahmanian

Enhancing Multimodal Continual Instruction Tuning with BranchLoRA (2025.acl-long)

Challenge:	Recent efforts to integrate low-rank adaptation (LoRA) with the Mixture-of-Experts (MoE) have achieved performance comparable to full-parameter fine-tuning by tuning much fewer parameters.
Approach:	They propose a parameter-efficient MoE method for low-rank adaptation with the Mixture-of-Experts (MoE) they use layers of LoRA experts to allocate more LoRA expert to middle layers .
Outcome:	The proposed method outperforms baseline models on six well-known NLP and commonsense QA benchmarks on LLAMA-2, Mistral, and Gemma.

Copied to clipboard

Duzhen Zhang, Yong Ren, Zhong-Zhi Li, Yahan Yu, Jiahua Dong, Chenxing Li, Zhilong Ji, Jinfeng Bai

Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Models (2022.coling-1)

Challenge:	Existing approaches to fine tune Multimodal Large Language Models (MLLMs) are prone to Catastrophic Forgetting (CF) existing approaches rely on the Mixture-of-Experts (MoE) LoRA framework to preserve previous instruction alignments.
Approach:	They propose an asymmetric tuning-freezing mechanism to mitigate parameter inefficiency . branch-specific routers are introduced to ensure optimal branch distribution over time .
Outcome:	The proposed framework outperforms existing frameworks on the latest MCIT benchmarks.

Copied to clipboard

Ze-Feng Gao, Peiyu Liu, Wayne Xin Zhao, Zhong-Yi Lu, Ji-Rong Wen

MMNMT: Modularizing Multilingual Neural Machine Translation with Flexibly Assembled MoE and Dense Blocks (2023.emnlp-main)

Challenge:	Recent results show that the mix-of-experts architecture is parameter inefficient . large-scale pre-trained language models can achieve excellent performance in many NLP tasks.
Approach:	They propose to build a parameter-efficient mix-of-experts architecture by sharing information across experts.
Outcome:	The proposed architecture increases model capacity without increasing computation costs.

Copied to clipboard

Shangjie Li, Xiangpeng Wei, Shaolin Zhu, Jun Xie, Baosong Yang, Deyi Xiong

Multilinguality

Challenge:	Mixture-of-Experts (MoE) based sparse architectures are prone to overfitting on low-resource language translation.
Approach:	They propose a modularized MNMT framework that flexibly assembles dense and MoE-based sparse modules to achieve the best of both worlds.
Outcome:	The proposed framework outperforms existing models on low-resource language translation and zero-shot translation on benchmark datasets.

CoGR-MoE: Concept-Guided Expert Routing with Consistent Selection and Flexible Reasoning for Visual Question Answering (2026.findings-acl)

Copied to clipboard

Xiyin Zeng, Yi Lu, Hao Wang

Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models (2024.acl-long)

Challenge:	Recent work has begun to address routing instability in VQA models by grouping similar concepts or routing based on examples.
Approach:	They propose a Concept-Guided Routing framework which incorporates semantics of the answer options to guide expert selection in the training phase.
Outcome:	The proposed framework delivers strong performance across multiple VQA tasks, demonstrating the effectiveness of the proposed framework.

Copied to clipboard

Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, Hongsheng Li

Advancing MoE Efficiency: A Collaboration-Constrained Routing (C2R) Strategy for Better Expert Parallelism Design (2025.naacl-long)

Challenge:	Mixture-of-Experts (MoE) LLMs achieve higher performance with fewer active parameters, but are still difficult to deploy due to their immense parameter sizes.
Approach:	They propose expert-level sparsification techniques to enhance the deployment efficiency of large language models by introducing plug-and-play expert pruning and skipping techniques.
Outcome:	The proposed methods reduce model sizes and increase inference speed while maintaining satisfactory performance across a wide range of tasks.

Copied to clipboard

Mohan Zhang, Pingzhi Li, Jie Peng, Mufan Qiu, Tianlong Chen

AdaTag: Multi-Attribute Value Extraction from Product Profiles with Adaptive Decoding (2021.acl-long)

Challenge:	Using Mixture-of-Experts, researchers have found that efficient MoE is difficult to achieve due to two key reasons: imbalanced expert activation and massive communication overhead.
Approach:	They propose a collaboration-constrained routing strategy that encourages more specialized expert groups and leverages expert specialization.
Outcome:	The proposed approach achieves an average performance improvement of 0.51% and 0.33% on LLaMA-MoE and Qwen-MaE respectively.

Copied to clipboard

Jun Yan, Nasser Zalmout, Yan Liang, Christan Grant, Xiang Ren, Xin Luna Dong

Efficiently Editing Mixture-of-Experts Models with Compressed Experts (2025.findings-emnlp)

Challenge:	Existing approaches to extract product attribute values are limited by knowledge sharing across different attributes.
Approach:	They propose to use adaptive decoding to handle extraction of product attribute values by parameterizing the decoder with pretrained attribute embeddings, through a hypernetwork and a Mixture-of-Experts module.
Outcome:	The proposed model is able to handle multiple attributes without sharing the entire network parameters across all attributes.

Copied to clipboard

Yifei He, Yang Liu, Chen Liang, Hany Hassan Awadalla

GuiLoMo: Allocating Experts and Ranks for LoRA-MoE via Bilevel Optimization with GuidedSelection Vectors (2025.findings-emnlp)

Challenge:	Mixture-of-Experts models allow for efficient scaling of large language models . fewer experts reduce computational costs, while more experts improve performance .
Approach:	They propose to activate only a subset of experts during training and inference . they propose compressed experts that preserve the most important experts .
Outcome:	The proposed approach preserves the most important experts while replacing other auxiliary activated experts with compressed experts.

Copied to clipboard

Xinrong Chen, Hengyuan Zhang, Yingmin Qiu, Xiao Liang, Ziyue Li, Guanyu Wang, Weiping Li, Tong Mo, Hayden Kwok-Hay So, Ngai Wong

DMoERM: Recipes of Mixture-of-Experts for Effective Reward Modeling (2024.findings-acl)

Challenge:	Low-Rank Adaptation (LoRA) methods are efficient for a large language model with reduced computational costs.
Approach:	They propose a layer-wise expert numbers and ranks allocation strategy with GuidedSelection Vectors.
Outcome:	The proposed method achieves superior or comparable performance to all baselines on three backbone models.

Copied to clipboard

Shanghaoran Quan

Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inference (2026.acl-long)

Challenge:	Using a reward model (RM) to improve the effectiveness of large language models, there are two challenges in training.
Approach:	They propose a reward model (RM) that is a proxy of human preferences and assigns scores to the outputs of the large language model (LLM) a human annotation consistency rate of 60% to 75% is causing training data to contain a lot of noise.
Outcome:	The proposed model outperforms state-of-the-art ensemble methods and mitigates the overoptimization problem.

Copied to clipboard

Baihui Liu, Kaiyuan Tian, Wei Wang, Zhaoning Zhang, Linbo Qiao, Dongsheng Li

HVGuard: Utilizing Multimodal Large Language Models for Hateful Video Detection (2025.emnlp-main)

Challenge:	Existing approaches that reduce expert activations lead to severe model performance degradation.
Approach:	They propose a framework that optimizes budget allocation coordinately at layer and token levels to minimize model performance degradation.
Outcome:	The proposed framework achieves 1.15 prefill and 1.34 decode speedups on DeepSeek-V2-Lite at half of the original budget.

Copied to clipboard

Yiheng Jing, Mingming Zhang, Yong Zhuang, Jiacheng Guo, Juan Wang, Xiaoyang Xu, Wenzhe Yi, Keyan Guo, Hongxin Hu

StableMoE: Stable Routing Strategy for Mixture of Experts (2022.acl-long)

Challenge:	Existing methods for hateful video detection rely on unimodal analysis or feature fusion . Existing tools struggle to capture cross-modal interactions and reason through implicit hate in sarcasm and metaphor .
Approach:	They propose a reasoning-based hateful video detection framework with multimodal large language models . they integrate Chain-of-Thought reasoning to enhance multimodal interaction modeling .
Outcome:	The proposed framework outperforms existing tools on two public datasets covering English and Chinese.

Copied to clipboard

Damai Dai, Li Dong, Shuming Ma, Bo Zheng, Zhifang Sui, Baobao Chang, Furu Wei

Challenge:	Existing learning-to-route methods suffer from the routing fluctuation issue . with the model scale growing, training speed will go slower and memory requirements are heavy .
Approach:	They propose a Mixture-of-Experts technique that can scale up the model size of Transformers with an affordable computational overhead.
Outcome:	The proposed method outperforms existing learning-to-route methods on language modeling and multilingual machine translation.

EchoVLM: Dynamic Mixture-of-Experts Vision-Language Model for Universal Ultrasound Intelligence (2026.acl-long)

Copied to clipboard

Chaoyin She, Ruifang Lu, Lida Chen, Wei Wang, Qinghua Huang

CartesianMoE: Boosting Knowledge Sharing among Experts via Cartesian Product Routing in Mixture-of-Experts (2025.naacl-long)

Challenge:	Ultrasound is the preferred early cancer screening modality due to non-ionizing radiation, cost-effectiveness, and real-time imaging.
Approach:	They propose to use ultrasound-tailored vision-language models with a mixture-of-experts architecture to train ultrasound-specific knowledge across seven anatomical systems.
Outcome:	The proposed model outperforms Qwen2-VL by 7.58 BLEU-1 and 3.45 ROUGE-1 points in report generation.

Copied to clipboard

Zhenpeng Su, Xing W, Zijia Lin, Yizhe Xiong, Minxuan Lv, Guangyuan Ma, Hui Chen, Songlin Hu, Guiguang Ding

Breaking ReLU Barrier: Generalized MoEfication for Dense Pretrained Models (2024.emnlp-main)

Challenge:	Large language models (LLMs) have been attracting much attention due to their impressive performance in all kinds of downstream tasks.
Approach:	They propose a mix-of-experts model that allows the model size to grow without raising training costs.
Outcome:	The proposed model outperforms existing models in perplexity and robustness tests.

Copied to clipboard

Jaeseong Lee, Seung-won Hwang, Wonpyo Park, Mingi Ji

Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts (2024.findings-emnlp)

Challenge:	Existing methods to convert pretrained dense models to MoEs are limited to ReLU-based models with natural sparsity.
Approach:	They propose a G-MoEfication approach for arbitrary dense models where activation sparsity assumptions no longer hold.
Outcome:	The proposed method reduces the inference cost associated with dense models by sparsely activating experts.

Copied to clipboard

Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, Tong Zhang

Explainability & Interpretability in NLP

Challenge:	Reinforcement learning from human feedback (RLHF) is the primary method for aligning large language models with human preferences.
Approach:	They propose to train an Absolute-Rating Multi-Objective Reward Model with multi-dimensional absolute-rating data.
Outcome:	The proposed model outperforms the LLM-as-a-judge method on RewardBench . it achieves state-of-the-art performance on the benchmark .

EAC-MoE: Expert-Selection Aware Compressor for Mixture-of-Experts Large Language Models (2025.acl-long)

Copied to clipboard

Yuanteng Chen, Yuantian Shao, Peisong Wang, Jian Cheng

Improved Sparse Upcycling for Instruction Tuning (2025.coling-main)

Challenge:	Mixture-of-Experts (MoE) has demonstrated promising potential in scaling LLMs . however, it is hindered by two critical challenges: substantial GPU memory consumption and low activated parameters.
Approach:	They propose an Expert-Selection Aware Compressor for Mixture-of-Experts (MoE) that aligns with the characteristics of MoE from the perspectives of quantization and pruning.
Outcome:	The proposed approach significantly reduces memory usage and improves inference speed with minimal performance degradation.

Copied to clipboard

Wangyi Jiang, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun

Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Merging (2025.acl-long)

Challenge:	Existing methods for sparse upcycling lead to performance degradation in instruction tuning scenarios.
Approach:	They propose a representation-based approach to convert dense language models into sparsely activated ones by initializing router weights from language models.
Outcome:	The proposed architecture improves model capabilities and routing consistency across multiple benchmarks.

Copied to clipboard

Tingfeng Hui, Zhenyu Zhang, Shuohuan Wang, Yu Sun, Hua Wu, Sen Su

MoKA:Parameter Efficiency Fine-Tuning via Mixture of Kronecker Product Adaption (2025.coling-main)

Challenge:	Existing methods for tuning large language models from dense to MoE face significant data requirements and require large-scale post-training.
Approach:	They propose an upcycling instruction tuning approach for tuning a dense pre-trained model into a MoE instruction model using genetic algorithm and parameter merging.
Outcome:	The proposed approach improves the performance of large language models with a small amount of seed data and improves their scaling.

Copied to clipboard

Beiming Yu, Zhenfei Yang, Xiushuang Yi

CoLA: Collaborative Low-Rank Adaptation (2025.findings-acl)

Challenge:	Low-Rank Adaptation (LoRA) is one of the most popular PEFT methods . low-rank update mechanism of LoRA somewhat limits its ability to approximate full-parameter fine-tuning during training process.
Approach:	They propose a parameter-efficient fine-tuning framework that combines Kronecker product with the Mixture-of-Experts method to achieve parameter efficiency and better model performance.
Outcome:	The proposed framework outperforms existing methods on the GLUE benchmark and instruction tuning tasks for large language models.

Copied to clipboard

Yiyun Zhou, Chang Yao, Jingyuan Chen

PROPER: A Progressive Learning Framework for Personalized Large Language Models with Group-Level Adaptation (2025.acl-long)

Challenge:	The scaling law of Large Language Models (LLMs) reveals diminishing return on performance as model scale increases.
Approach:	They propose a more flexible LoRA architecture with an efficient initialization scheme . they propose combining three collaborative strategies to enhance performance .
Outcome:	The proposed model outperforms existing methods in low-sample scenarios.

Copied to clipboard

Linhai Zhang, Jialong Wu, Deyu Zhou, Yulan He

SliceMoE: Routing Embedding Slices Instead of Tokens for Fine-Grained and Balanced Transformer Scaling (2025.emnlp-main)

Challenge:	Personalized large language models (LLMs) aim to tailor outputs to user preferences . however, user data is typically sparse, making it challenging to adapt LLMs to specific user patterns.
Approach:	They propose a progressive learning framework that groups users based on preferences and adapts LLMs in stages.
Outcome:	The proposed approach outperforms SOTA models across multiple tasks.

Copied to clipboard

Harshil Vejendla

BTW: A Non-Parametric Variance Stabilization Framework for Multimodal Model Integration (2025.findings-emnlp)

Challenge:	Token-level routing assigns an entire semantic spectrum to each expert, creating capacity bottlenecks, load-balancing pathologies, and limited specialisation.
Approach:	They propose an architecture that routes contiguous slices of a token’s hidden vector and a lightweight shared router predicts the top-k experts.
Outcome:	The proposed architecture achieves 1.7x faster inference than dense baselines, 12–18% lower perplexity than parameter-matched token-MoE, and improved expert balance.

Copied to clipboard

Jun Hou, Le Wang, Xuan Wang

Text Classification

Challenge:	Existing methods for multimodal learning are difficult to scale beyond two modalities and lack resolution for instance-level control.
Approach:	They propose a bi-level weighting framework that combines instance-level Kullback-Leibler divergence and modality-level mutual information to dynamically adjust modality importance during training.
Outcome:	The proposed method significantly improves regression performance and multiclass classification accuracy.

Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understanding (2023.findings-emnlp)

Copied to clipboard

Yu Zhang, Hao Cheng, Zhihong Shen, Xiaodong Liu, Ye-Yi Wang, Jianfeng Gao

TalkLoRA: Communication-Aware Mixture of Low-Rank Adaptation for Large Language Models (2026.acl-long)

Challenge:	Pre-trained language models (LMs) have shown effectiveness in literature understanding tasks, especially when tuned via contrastive learning.
Approach:	They propose a multi-task contrastive learning framework that enables common knowledge sharing across different scientific literature understanding tasks while preventing task-specific skills from interfering with each other.
Outcome:	The proposed framework outperforms state-of-the-art pre-trained language models on a comprehensive dataset.

Copied to clipboard

Lin Mu, Haiyang Wang, Li Ni, Lei Sang, Zhize Wu, Peiquan Jin, Yiwen Zhang

From Pseudo-Balancing to True Specialization: Memory-Aware Routing for Mixture-of-Experts (2026.findings-acl)

Challenge:	Existing LoRA methods assume that experts operate independently, leading to unstable routing, expert dominance.
Approach:	They propose a communication-aware MoELoRA framework that relaxes this assumption by introducing expert-level communication prior to routing.
Outcome:	The proposed framework outperforms vanilla LoRA and MoELoRA on diverse language understanding tasks while maintaining expert dominance.

Copied to clipboard

Peixuan Hou, Yunbo Hou, Bin Chen, LI He, Jian Xu, Weiping Li, Bo Zheng, Guojie Song

Less, but Better: Efficient Multilingual Expansion for LLMs via Layer-wise Mixture-of-Experts (2025.acl-long)

Challenge:	Existing methods to optimize expert-centered load balancing fail to account for pseudo-balance phenomenon . severe knowledge overlap among experts leads to redundant representations and inefficient parameter utilization .
Approach:	They propose a method that prioritizes expert utilization over semantic alignment . they use memory-aware routing to ensure expert load balancing is consistent .
Outcome:	Experimental results show that MAR improves expert specialization by 35% and accuracy by 2%-25% . MAR matches baseline performance with only half the experts .

Copied to clipboard

Xue Zhang, Yunlong Liang, Fandong Meng, Songming Zhang, Yufeng Chen, Jinan Xu, Jie Zhou

LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-Training (2024.emnlp-main)

Challenge:	Existing large language models (LLMs) have remarkable ability in high-resource languages, but their performance in multilingual scenarios is still limited.
Approach:	They propose a layer-wise expert allocation algorithm to determine the appropriate number of new experts for each layer.
Outcome:	The proposed method outperforms the previous state-of-the-art baseline with 60% fewer experts in the single-expansion setting and 33.3% fewer in the lifelong-expanding setting.

Copied to clipboard

Tong Zhu, Xiaoye Qu, Daize Dong, Jiacheng Ruan, Jingqi Tong, Conghui He, Yu Cheng

SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations (2023.acl-long)

Challenge:	Mixture-of-Experts (MoE) has gained increasing popularity as a framework for scaling up large language models.
Approach:	They investigate how to build Mixture-of-Experts (MoE) models from existing large language models . they use expert construction, Continual pre-training and data sampling strategies .
Outcome:	The proposed model outperforms existing models with similar parameters on a wide range of tasks.

Copied to clipboard

Paul-Ambroise Duquenne, Hongyu Gong, Ning Dong, Jingfei Du, Ann Lee, Vedanuj Goswami, Changhan Wang, Juan Pino, Benoît Sagot, Holger Schwenk

Towards A Unified View of Sparse Feed-Forward Network in Pretraining Large Language Model (2023.emnlp-main)

Challenge:	SpeechMatrix is a large-scale multilingual corpus of speech-to-speech translations mined from real speech of European Parliament recordings.
Approach:	They present a large-scale multilingual corpus of speech-to-speech translations mined from real speech of European Parliament recordings.
Outcome:	The proposed model can train bilingual models on 136 language pairs with 418 thousand hours of speech.

Copied to clipboard

Zeyu Liu, Tim Dettmers, Xi Lin, Veselin Stoyanov, Xian Li

Dynamic Expert Specialization: Towards Catastrophic Forgetting-Free Multi-Domain MoE Adaptation (2025.emnlp-main)

Challenge:	Large and sparse feed-forward layers (S-FFN) have proven effective in scaling up the model size for pretraining large language models.
Approach:	They compare S-FFN architectures for language modeling and compare their performance and efficiency . they found a simpler selection method that selects blocks through their mean aggregated hidden states .
Outcome:	The proposed model size and selection method achieve lower perplexity in language model pretraining compared to existing MoE architectures.

Copied to clipboard

Junzhuo Li, Bo Wang, Xiuze Zhou, Xuming Hu

DIVE into MoE: Diversity-Enhanced Reconstruction of Large Language Models from Dense into Mixture-of-Experts (2025.acl-long)

Challenge:	Existing approaches to adapt Mixture-of-Experts models to multiple domains are prohibitive computation, cross-domain interference or require separate runs per domain.
Approach:	They propose a dynamic expert specialization framework for multi-domain adaptation of Mixture-of-Experts models.
Outcome:	The proposed framework reduces forgetting by 89% compared to full fine-tuning as domains scale from 2 to 6 and achieves faster convergence than conventional methods.

Copied to clipboard

Yuchen Feng, Bowen Shen, Naibin Gu, Jiaxuan Zhao, Peng Fu, Zheng Lin, Weiping Wang

Sparsifying Mamba (2025.findings-emnlp)

Challenge:	Existing methods for reconstruction of large language models overlook diversity among experts, leading to potential redundancy.
Approach:	They propose a pruning-based expert reconstruction method that prunes a specific LLM and retrains it on routers, experts and normalization modules.
Outcome:	The proposed method outperforms pruning and MoE reconstruction methods on Llama-style models with open-source training corpora.

Copied to clipboard

An Wang, Ruobing Xie, Shuaipeng Li, Xingwu Sun, Zhanhui Kang

MoA: Heterogeneous Mixture of Adapters for Parameter-Efficient Fine-Tuning of Large Language Models (2026.acl-long)

Challenge:	Existing attempts to integrate sparsification with Mamba fail to leverage Mamba's internal structure for fine-grained sparsifying.
Approach:	They propose to use Mamba to integrate sparsification into Mamba and propose a flexible and effective mechanism for parameter scalability.
Outcome:	The proposed framework can independently achieve parameter scalability and has stronger performance.

Copied to clipboard

Jie Cao, Tianwei Lin, Bo Yuan, Rolan Yan, Hongyang He, Wenqiao Zhang, Juncheng Li, Dongping Zhang, Siliang Tang, Yueting Zhuang

Specialization without Sparsity: Efficient and Expressive Split-Path Experts for LLM Fine-Tuning (2026.findings-acl)

Challenge:	Existing methods for parameter-efficient fine-tuning (PEFT) are limited by computational costs and performance degradation.
Approach:	They propose a method that integrates Low-Rank Adaptation and Mixture-of-Experts (MoE) they propose combining expert load imbalance and representation collapse to improve LLM performance .
Outcome:	The proposed method outperforms homogeneous MoE-LoRA architectures in performance and parameter efficiency.

Copied to clipboard

Ganghao Liu, Qin Zhou, Zhe Wang, Xuehan Lu, Haihua Huang, Yunfei Tong, Heng Tian

Profiling-Free Mixed-Precision Quantization for MoE LLMs via Fuzzy Rule Interpolation (2026.acl-long)

Challenge:	Parameter-efficient fine-tuning (PEFT) is a low-cost alternative to full fine-timing due to the massive overhead.
Approach:	They propose a Mixture-of-Experts approach that enhances specialization while maintaining low resource overhead.
Outcome:	The proposed approach outperforms or matches state-of-the-art methods on GLUE, GSM8K, MBPP, and a text rewriting task from SmolTalk.

Copied to clipboard

Huachen Qi, Ruiyu Zhuo, Bowen Shi, Xiang Chang, Fei Chao, Changjing Shang, Qiang Shen

Mixture-of-LoRAs: An Efficient Multitask Tuning Method for Large Language Models (2024.lrec-main)

Challenge:	Large Language Models are scaling in size and capability, driving substantial computational and memory costs.
Approach:	They propose a mixed-precision quantization framework that uses fuzzy rule interpolation to predict quantization error from only sparse samples.
Outcome:	The proposed framework accelerates the profiling phase by up to 15.7 on DeepSeek-V2 while achieving comparable or slightly superior zero-shot accuracy.

Copied to clipboard

Wenfeng Feng, Chuzhan Hao, Yuewei Zhang, Yu Han, Hao Wang

LightMoE: Task-Aware Expert Availability Management for Memory-Efficient MoE-LLM Inference (2026.findings-acl)

Challenge:	Instruction Tuning has the potential to stimulate or enhance specific capabilities of large language models.
Approach:	They propose a mixture-of-LoRAs architecture which is a parameter-efficient tuning method designed for multi-task learning with LLMs.
Outcome:	The proposed method can be iteratively adapted to a new domain, enabling quick domain-specific adaptation.

Copied to clipboard

Puhan Luo, Yunhao Yao, Junyang Wang, Junyang Zhang, Xiangyang Li

Decoding Knowledge Attribution in Mixture-of-Experts: A Framework of Basic-Refinement Collaboration and Efficiency Analysis (2025.acl-long)

Challenge:	Existing solutions for balancing model accuracy with inference latency are limited due to memory constraints.
Approach:	They propose a framework for memory-efficient MoE inference that exploits the functional redundancy and temporal locality of expert activation.
Outcome:	The proposed framework improves accuracy-efficiency trade-off by 4.3% over pruned models and 2.4% over dynamic swapping methods while maintaining inference latency comparable to pruned model.

Copied to clipboard

Junzhuo Li, Bo Wang, Xiuze Zhou, Peijie Jiang, Jia Liu, Xuming Hu

DRAE: Dynamic Retrieval-Augmented Expert Networks for Lifelong Learning and Task Adaptation in Robotics (2025.acl-long)

Challenge:	Existing attribution methods for dense models fail to capture dynamic routing-expert interactions in sparse MoE architectures.
Approach:	They propose to analyze sparse MoE architectures against dense models to capture dynamic routing-expert interactions.
Outcome:	The proposed algorithm shows that sparse models achieve higher efficiency per layer . it also shows that deep Qwen-MoE mitigates expert failures while minimizing complexity .

Copied to clipboard

Yayu Long, Kewei Chen, Long Jin, Mingsheng Shang

AlphaLoRA: Assigning LoRA Experts Based on Layer Training Quality (2024.emnlp-main)

Challenge:	Experimental results show that Dynamic Retrieval-Augmented Expert Networks outperforms baseline approaches in long-term task retention and knowledge reuse.
Approach:	They propose a dynamic routing architecture that leverages MoE and Retrieval-Augmented Generation to augment the learning process.
Outcome:	The proposed architecture outperforms baseline approaches in long-term task retention and knowledge reuse.

Copied to clipboard

Peijun Qing, Chongyang Gao, Yefan Zhou, Xingjian Diao, Yaoqing Yang, Soroush Vosoughi

Stress-Testing Emotional Support Models: Moving from Homogeneous to Diverse Help Seekers (2026.findings-acl)

Challenge:	Recent studies combine LoRA with Mixture-of-Experts (MoE) to improve performance in Large Language Models.
Approach:	They propose a method to combine LoRA and Mixture-of-Experts (MoE) to improve performance in Large Language Models.
Outcome:	The proposed method reduces redundancy in LoRA experts within the MoE architecture, and improves training quality across layers.

Copied to clipboard

Chaewon Heo, Cheyon Jin, Yohan Jo

Dialogue Systems & Conversational Agents

Challenge:	Existing simulators fail to capture behavioral diversity of real-world seekers . lack of reliable automated evaluation frameworks hinders field's establishment .
Approach:	They propose a controllable seeker simulator driven by nine psychological and linguistic features that underpin seeker behavior.
Outcome:	The proposed model achieves superior profile adherence and behavioral diversity compared to existing approaches.

CLIP-UP: A Simple and Efficient Mixture-of-Experts CLIP Training Recipe with Sparse Upcycling (2025.findings-emnlp)

Copied to clipboard

Xinze Wang, Chen Chen, Yinfei Yang, Hong-You Chen, Bowen Zhang, Aditya Pal, Xiangxin Zhu, Xianzhi Du

Towards Stable and Effective Reinforcement Learning for Mixture-of-Experts (2026.acl-long)

Challenge:	Mixture-of-Experts (MoE) models are crucial for scaling model capacity while controlling inference costs.
Approach:	They propose an alternative training strategy that converts a dense CLIP model into a sparse MoE architecture.
Outcome:	The proposed training strategy outperforms dense models on COCO and Flickr30k benchmarks.

Copied to clipboard

Di Zhang, Xun Wu, Shaohan Huang, Lingjie Jiang, Yaru Hao, Li Dong, Zewen Chi, Zhifang Sui, Furu Wei

Reasoning

Challenge:	Reinforcement learning with verifiable rewards (RLVR) training with Mixture-of-Experts policies remains fragile and prone to reward collapse.
Approach:	They propose a router shift-based policy optimization method that computes a per-token router-shift ratio conditioned on the previously activated experts and applies stop-gradient and a lower-bound floor.
Outcome:	The proposed method achieves better performance and greater stability than previous methods.

AEA: Adaptive Expert Allocation Improves Sentence Embeddings from Mixture-of-Experts LLM (2026.acl-long)

Copied to clipboard

Shufan Yang, Zifeng Cheng, Zhiwei Jiang, Qingfeng Qi, Yafeng Yin, Cong Wang, Ao Zhou, Qing Gu

SAMoRA: Semantic-Aware Mixture of LoRA Experts for Task-Adaptive Learning (2026.findings-acl)

Challenge:	Existing methods to improve embeddings from Mixture-of-Experts models allocate a fixed number of experts uniformly across all layers and tokens, ignoring inter-layer and inter-token heterogeneity.
Approach:	They propose an Adaptive Expert Allocation framework that performs layer-wise and token-wise expert allocation to enhance embedding quality.
Outcome:	The proposed method improves embedding quality across multiple MoE models.

Copied to clipboard

Boyan Shi, Wei Chen, Shuyuan Zhao, Junfeng Shen, Shengnan Guo, Shaojiang Wang, Huaiyu Wan

Too Helpful, Too Harmless, Too Honest or Just Right? (2025.emnlp-main)

Challenge:	Existing methods for multitask learning fail to match input semantics with expert capabilities, leading to weak expert specialization.
Approach:	They propose a parameter-efficient mixture-of-experts framework for task-adaptive learning that aligns textual semantics with the most suitable experts for precise routing.
Outcome:	The proposed framework outperforms the state-of-the-art methods and holds excellent task generalization capabilities.

Copied to clipboard

Gautam Siddharth Kashyap, Mark Dras, Usman Naseem

Cross-MoE: An Efficient Temporal Prediction Framework Integrating Textual Modality (2025.emnlp-main)

Challenge:	Existing methods optimize for individual alignment dimensions in isolation, leading to trade-offs and inconsistent behavior.
Approach:	They propose a modular alignment framework that integrates a Mixture of Calibrated Experts (MoCaE) within the Transformer architecture.
Outcome:	The proposed framework outperforms baselines on three alignment benchmarks, achieving 32.5% win rate, 33.9% safety score, and 28.4% truthfulness.

Copied to clipboard

Ruizheng Huang, Zhicheng Zhang, Yong Wang

COMPEL: Compensated Mixture-of-Experts Pruning with Expert-Layer distribution (2026.findings-acl)

Challenge:	Existing models ignore dynamic and different relations between time series patterns and textual features, which leads to poor performance in temporal-textual feature fusion.
Approach:	They propose a temporal-textual fusion framework that replaces Cross Attention with Cross-Ranker to reduce computational complexity and enhances modality-aware correlation memorization with Mixture-of-Experts (MoE) networks to tolerate the distributional shifts in time series.
Outcome:	The proposed framework reduces MSE by 8.78% compared to the current SOTA model and requires only 75% of computational overhead and 12.5% of activated parameters.

Copied to clipboard

Seohee Yoon, Yong Suk Choi

Towards Identification and Intervention of Safety-Critical Parameters in Large Language Models (2026.findings-acl)

Challenge:	Mixture-of-Experts (MoE) architectures are effective for scaling Large Language Models (LLMs) however, existing pruning methods adopt uniform pruning across layers, which fails to capture layer-wise variations in expert importance and redundancy.
Approach:	They propose a Mixture-of-Experts pruning method that activates only a subset of experts during inference by estimating expert importance using Fisher information.
Outcome:	The proposed pruning method outperforms existing pruning methods while reducing inference latency and peak GPU memory usage.

Copied to clipboard

Weiwei Qi, Zefeng Wu, Tianhang Zheng, Zikang Zhang, Xiaojun Jia, Zhan Qin, Kui Ren

Union-of-Experts: Neurons in Mixture-of-Experts are Secretly Routers (2026.acl-long)

Challenge:	Existing safety-related methodologies for large language models are lacking . despite advances in safety alignment techniques, safeguarding LLMs during adaptation to various tasks remains a challenge.
Approach:	They propose a framework to quantify how different parameters affect LLM safety . they propose two targeted intervention paradigms for safety enhancement and preservation .
Outcome:	The proposed framework reveals safety-critical patterns across different LLM architectures.

Copied to clipboard

Songhao Wu, Ang Lv, Ruobing Xie, Samm Sun, Di Wang, Rui Yan, Yankai Lin

Uncertainty-Aware Routing for Principled Alignment with MoE Dynamics (2026.acl-long)

Challenge:	Mixture-of-Experts (MoE) models rely on an external router to assign tokens to experts, resulting in suboptimal performance.
Approach:	They propose an MoE variant that performs "expert-autonomous routing" by pre-designating a fraction of neurons within each expert as "routing neurons" they pre-train UoE models with up to 3B parameters and show they outperform traditional MoEs with matched efficiency.
Outcome:	The proposed model outperforms existing models with 3B parameters and provides valuable insights into expert-autonomous selection and the broader routing mechanisms of MoE models.

Copied to clipboard

Yilong Chen, Junyuan Shang, Yuchen Feng, Zhenyu Zhang, Naibin Gu, Ziqi Wang, Tingwen Liu, Shuohuan Wang, Yu Sun, Hua Wu, Haifeng Wang

Challenge:	Mixture-of-Experts (MoE) is a cornerstone for scaling LLMs, yet its training dynamics remain poorly understood, often leading to sub-optimal specialization.
Approach:	They propose to use Helmholtz Free Energy and Router Entropy to study the MoE lifecycle and identify a universal Three-Stage Phase Transition .
Outcome:	The proposed model reduces perplexity and improves expert distinctiveness, offering a principled path toward thermodynamically aligned computation.

Improved Policy Optimization for Mixture-of-Experts Models: Importance Sampling and Rewarding from an Expert-Centric Perspective (2026.findings-acl)

Copied to clipboard

Yining Qian, Jinpeng Li, Fei Mi, Lifeng Shang, Xiang Zhang

Challenge:	Existing approaches to reinforcement learning (RL) suffer from training instability . existing approaches often ignore token-specific discrepancies in expert assignments .
Approach:	They propose to introduce expert-level importance sampling to reduce complexity of RL . they propose to leverage expert-centric granularity to ensure a rigorous alignment between reward signals and policy updates.
Outcome:	The proposed method outperforms strong baselines across reasoning tasks.

Truth as a Trajectory: What Internal Representations Reveal About Large Language Model Reasoning (2026.acl-long)

Copied to clipboard

Hamed Damirchi, null Imezadelajara, Ehsan Abbasnejad, Afshar Shamsi, Zhen Zhang, Javen Qinfeng Shi