Mixture of insighTful Experts (MoTE): The Synergy of Reasoning Chains and Expert Mixtures in Self-Alignment

Challenge:	Recent studies show that reasoning abilities contribute significantly to model safety, while integrating Mixture-of-Experts (MoE) architectures can further enhance alignment.
Approach:	They propose a framework that synergistically combines reasoning chains and expert mixtures to improve self-alignment.
Outcome:	The proposed framework improves model safety, jailbreak resistance, and over-refusal capabilities, achieving performance comparable to OpenAI’s state-of-the-art o1 model.

Similar Papers

Overview

Challenge

Approach

Outcome

Bag of Tricks for Sparse Mixture-of-Experts: A Benchmark Across Reasoning, Efficiency, and Safety (2025.findings-emnlp)

Copied to clipboard

Mufan Qiu, Zheyu Shen, Pingzhi Li, Ang Li, Tianlong Chen

Responsible & Trustworthy NLP

Challenge:	Existing benchmarks focus on isolated aspects of MoE, with conflicting conclusions . a lack of consensus on optimal design choices is limiting to specific aspects of the model.
Approach:	They propose to evaluate two popular MoE backbones across four dimensions of design choices . they find token-level routing and z-loss regularization improve reasoning performance .
Outcome:	The proposed framework evaluates two popular MoE backbones on over eight metrics.

A Closer Look into Mixture-of-Experts in Large Language Models (2025.findings-naacl)

Copied to clipboard

Ka Man Lo, Zeyu Huang, Zihan Qiu, Zili Wang, Jie Fu

Language Models

Challenge:	Mixture-of-experts (MoE) architectures are gaining increasing attention for their unique properties and remarkable performance.
Approach:	They propose a mixture-of-experts architecture that allows for model scaling without sacrificing computational efficiency.
Outcome:	The proposed model increases model size without sacrificing computational efficiency . the proposed model is modular and can be used by a broad spectrum of practitioners .

Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modules (2026.findings-eacl)

Copied to clipboard

Yilun Liu, Yunpu Ma, Yuetian Lu, Shuo Chen, Zifeng Ding, Volker Tresp

Language Models

Challenge:	Existing Parameter-Efficient Fine-Tuning (PEFT) strategies that focus on specialized experts are not effective for Mixture-of-Experts (MoE).
Approach:	They propose to integrate a dynamic routing mechanism among specialized experts in Mixture-of-Experts (MoE) .
Outcome:	Extensive experiments on commonsense and math reasoning tasks validate the performance and efficiency of the proposed routed approach.

MoA: Heterogeneous Mixture of Adapters for Parameter-Efficient Fine-Tuning of Large Language Models (2026.acl-long)

Copied to clipboard

Jie Cao, Tianwei Lin, Bo Yuan, Rolan Yan, Hongyang He, Wenqiao Zhang, Juncheng Li, Dongping Zhang, Siliang Tang, Yueting Zhuang

Semantic Text Processing

Challenge:	Existing methods for parameter-efficient fine-tuning (PEFT) are limited by computational costs and performance degradation.
Approach:	They propose a method that integrates Low-Rank Adaptation and Mixture-of-Experts (MoE) they propose combining expert load imbalance and representation collapse to improve LLM performance .
Outcome:	The proposed method outperforms homogeneous MoE-LoRA architectures in performance and parameter efficiency.

MixtureKit: A General Framework for Composing, Training, and Visualizing Mixture-of-Experts Models (2026.acl-demo)

Copied to clipboard

Ahmad Chamma, Omar El Herraoui, Guokan Shang

Language Models

Challenge:	MixtureKit is a modular open-source framework for constructing, training, and analyzing Mixture-of-Experts (MoE) models from arbitrary pre-trained or fine-tuned checkpoints.
Approach:	They propose a modular open-source framework for constructing, training, and analyzing Mixture-of-Experts (MoE) models from arbitrary pre-trained or fine-tuned checkpoints.
Outcome:	Experiments on multilingual code-switched (Arabic–Latin) show that BTX models built with MixtureKit outperform dense baselines across multiple benchmarks.

From Outcome to Process: Optimizing MoE Load Balancing with MCTS (2026.findings-acl)

Copied to clipboard

Wenjun Ke, Hengyuan Xu, Ziyu Shang, Yao He, Jiahao Wang, Zijie Xu, Peng Wang, Yuhang Lou, Jiajun Liu

Green & Sustainable NLP

Challenge:	Existing balancing strategies focus on constraining the final distribution of expert usage, but overlook the routing decisions made at each layer.
Approach:	They propose a three-stage framework that leverages process-level rewards to guide balanced expert routing.
Outcome:	Extensive experiments show that LayerMoE improves the performance of state-of-the-art LoRA-MoA baselines, yielding an average accuracy gain of 1.39%.

Getting MoRE out of Mixture of Language Model Reasoning Experts (2023.findings-emnlp)

Copied to clipboard

Chenglei Si, Weijia Shi, Chen Zhao, Luke Zettlemoyer, Jordan Boyd-Graber

Responsible & Trustworthy NLP

Challenge:	Existing large language models (LLMs) have poor generalizability on question types beyond those seen in the prompt.
Approach:	They propose a framework that integrates specialized language models to generalize across question types that require distinct reasoning abilities.
Outcome:	The proposed framework gives higher accuracy than any single specialized model on a collection of 12 QA datasets from four reasoning types.

LoRACoE: Improving Large Language Model via Composition-based LoRA Expert (2025.emnlp-main)

Copied to clipboard

Guanyu Li, Zhiheng Xi, Zhihao Zhang, Boyang Hong, Tao Gui, Qi Zhang, Xuanjing Huang

Language Models

Challenge:	Recent studies show that the Mixture of Experts architecture improves performance of large language models.
Approach:	They propose a method to build static experts using LoRA parameters . they propose to use rank-level parameters to build experts based on rank-based parameters based in LoRA module.
Outcome:	The proposed method improves task performance across a broader range of tasks.

Part-Of-Speech Sensitivity of Routers in Mixture of Experts Models (2025.coling-main)

Copied to clipboard

Elie Antoine, Frederic Bechet, Phillippe Langlais

Tagging

Challenge:	a study examines the behavior of routers in Mixture of Experts (MoE) models . experts with similar linguistic traits are often routed to the same expert regardless of context .
Approach:	They investigate how tokens are routed based on their linguistic features . they aim to explore whether experts specialize in processing tokens with similar linguistic traits .
Outcome:	The proposed model-integrated routers are based on Mixture of Experts (MoE) models . the results show that expert specialization is high for POS categories .

MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs (2025.naacl-long)

Copied to clipboard

Yuhang Zhou, Giannis Karamanolakis, Victor Soto, Anna Rumshisky, Mayank Kulkarni, Furong Huang, Wei Ai, Jianhua Lu

Language Models

Challenge:	State-of-the-art methods for merging expert models with different architectures do not address parameter interference and require extensive fine-tuning to restore performance.
Approach:	They propose a method for merging experts with different architectures into a unified Mixture-of-Experts model with a goal of enhancing performance in each domain while retaining effectiveness on general tasks.
Outcome:	Experiments across multiple domains show that the proposed methods reduce fine-tuning costs and improve performance over state-of-the-art methods.